Record Filter¶

Supports filtering of input data based on user-configured expressions.

Configuration¶

The configuration tabs for this stage are General, Basic and Input/Output.

General¶

Name	Required?	Description
Name	Yes	The name of the stage.
Description	No	The description of the stage.
Stage Library	Yes	The streaming operator library to which the stage belongs.
Required Fields	No	The fields that the data records must contain. If the specified fields are not included, the record will be filtered out.
Preconditions	No	The conditions that must be satisfied by the data records. Records that do not meet the conditions will be filtered out. For example, `${record:value('/value') > 0}`. For the syntax of EL expressions, see Expression Language.
On Record Error	Yes	The processing method for error data. Discard: Error data will be discarded and ignored Send to Error: Error messages will be reported Stop Pipeline: The pipeline will be stopped

Basic¶

Name	Required?	Description
Quality Filter	No	Filter the data according to the data quality. Only records that meet the quality conditions will be processed by this stage.
Filter Expression	Yes	The data is filtered according to the configured expression, see below for the specific expression syntax: Filter Expression Syntax Summary
Output Measurement ID	Yes	Note: If a data satisfies multiple Filter Expressions, the record will be output multiple times and the mesurementId of the output record will be replaced with the configured Output Measurement ID. replaced accordingly

Filter Expression Syntax Summary¶

To support more flexible and convenient filtering of data, the Filter Expression configuration supports the use of Aviator Script to filter the input data. Here is a brief description of the syntax and functions for this operator usage scenario, for further details, please refer to Aviator Script Documentation.

Basic Types and Operations¶

Type	Description	Sample
Number	Can be integer, floating point, etc.	123，3.1415
String	Consecutive characters enclosed in single or double quotation marks	‘hello’，”world”
+, -, *, /	Add, subtract, multiply and divide, + can also be used to splice strings	1+1.2, 5/2, ‘ab’ + ‘c’
<, <=, >, >=, ==, !=	Comparison Operators	3 != 1
&&, \|\|, !	Logical with/or/without operators	true && false

Variables¶

Generally, variables are not created directly in the Filter Expression configuration; the Record Formatter algorithm already converts the input data into an object with the name record for use.

Built-in Objects	Description
record	The input data is converted into a Map object to be passed into the expression execution context, which can be accessed by record as a variable name. Note: For a batch of input data, record represents only one of the data, and the expression will act on each record of the batch in turn.

Since record is a Map object, Aviator provides a syntax for accessing the contents of the Map object in addition to using functions. Here is a brief example.

Syntax	Description
record.a.b	Indicates access to the value of the b attribute in the a attribute on the variable record
#record.a[1].b	indicates access to the b attribute value of the 2nd (starting from 0) object in the array a on the variable record, note: if the variable in the access path is an element of the array/List, you need to start with the #
#`record.name:a.b`	indicates access to the value of the b attribute in the name:a attribute on the variable record. Note: If the name of the accessed attribute/variable contains special characters, it needs to be surrounded by # plus ``

Common Functions¶

There are many functions built into Aviator Script, here are only the common functions used to configure Filter Expression, for more functions please refer to Aviator Script Documentation.

Functions	Description
seq.contains_key(map, key)	Get the value of the corresponding element from a list, array or map, where element is the index position of the element (starting from 0) for lists and arrays, and element is the key for maps
seq.get(map, element)	Returns true when there is a key in the map (which may be null); for arrays and chained tables, the key can be index, and returns true when index is in the valid range [0..len-1], otherwise returns false
string.startsWith(s1,s2)	Returns Boolean if string s1 starts with string s2.
string.endsWith(s1,s2)	If or not the string s1 ends with the string s2,return Boolean

Sample¶

For a record like the following:

{
    "orgId": "o15638787142121",
    "modelId": "Darwen_SGBuilding",
    "modelIdPath": "/Darwen_Building/Darwen_SGBuilding",
    "assetId": "UXQB9oLc",
    "pointId": "temp",
    "time": 1542609276270,
    "value": 23.4,
    "quality": 0,
    "dq": 0,
    "attr": {},
    "measurementId": "Darwen_SGBuilding::temp",
    "assetTags": {
        "DCMModel": {
            "DcmModel:Darwen_Building": null,
            "DcmModel:Darwen_SGBuilding": null
        }
    },
    "measurementTags": {
        "MyHaystack": {
            "MyHaystack:zone": {"floor": 2},
            "MyHaystack:co2": null,
            "MyHaystack:sensor": null
        }
    }
}

Filter Expression Sample	Description
`seq.contains_key(record.assetTags.DCMModel, 'DcmModel:Darwen_SGBuilding') && #record.measurementTags.MyHaystack.MyHaystack:zone.floor > 1`	First, determine whether record.assetTags.DCMModel This Map object contains a Key with the name DcmModel:Darwen_SGBuilding, if true, continue to determine whether record.measurementTags.MyHaystack: zone.floor is greater than 1, both are true, then return true

Output Results¶

Output Example¶

Configuration information