Introduction

Filtering can be used to process the incoming data before it is sent to the outputs, allowing flexible data manipulation. Unlike other data manipulation operations (scripting and flattening), filtering acts on the data schema rather than the payload. That is, filtering acts on an object representation rather than on its instance.

For example, given the incoming data below:

Filtering it to remove "id" and "UUID" results in the data held in the datacenter to be:

On subsequent data, should the array contain more entries and assuming the same filtering is applied, then the Datacenter only reads the property "label" for all entries.

Currently, the following providers support filtering:

These are the following topics:

• Introduction

• Accessing the Filter Page

• Interacting with the Tree: Filtering Nodes

• Searching Nodes and Inspecting Sample Values

Accessing the Filter Page

To access the filtering page, select a dataset of one of the supported providers, hover on an instance of a template and in the hamburger menu that appears, select Filter:

This opens a dialog, where the "schema" (an abstract representation that describes your existing data format) is shown as an interactive tree.

Interacting with the Tree: Filtering Nodes

The tree shown in the filtering dialog can be panned and zoomed: use your mouse's left-button to pan, and the mouse scroll wheel to zoom.

A node in the tree can be in one of the following three states:

• Active: The node is added to the dataset data. If a node represents an object, its children are also added if no further filtering is applied, or the individual filtering on the children is considered.

• Inactive: The node is not added to the dataset data. If a node represents an object, all its children is excluded from the dataset.

• Disabled: No interaction is possible with the node.

When adding a new data source, by default, the incoming data is set to active. That is, all fields in the incoming object are added to the dataset data. To start filtering nodes in the tree, click on the nodes you wish to add to the dataset.

When clicking on a node, there are three possible scenarios:

1. A leaf node that is directly connected to the root is clicked: a single node is added to the dataset.
   

   

2. A leaf node that is a children of an object is clicked: the leaf node, its parents, and all other parents up to the root are added to the dataset.
   

   

3. A node representing an object is clicked: the object node, its children, and its parents and all other parents up to the root are added to the dataset.
   

   

Once you start filtering a dataset, you can click on a node to toggle its state. That is, clicking on a node for the first time sets its state to active. Clicking on it again sets its state to inactive.

To further illustrate how filtering works, consider the following filtered tree:

This selection results in the dataset to hold the following (flattened) values.

Searching Nodes and Inspecting Sample Values

Nodes in the tree can be searched using the Search nodes field in the top-left of the dialog. Start typing a node name in the text field to see all possible matches. By selecting one of the results, the corresponding node briefly changes color and size.

To get an idea of the incoming data, sample payload can be inspected for each node in the tree. To reveal the sample content of a node, hover on its name.