Filter Steps Reference

This page contains a thorough reference for each of the 12 transformation steps that the EventProcessor parsing pipeline offers.

1. json

Reads a JSON string and retrieves its keys.

• Fields: source (Required), where (Optional CEL expression).

• Placement: All extracted keys are automatically prefixed with log. (e.g., {"id": 1} becomes log.id: 1).

Example:

- json: 
    source: raw
    where: 'contains(raw, "{")'

2. rename

Assigns new names to existing fields.

• Fields: from (Array of source paths), to (Target path), where (Optional).

• Behavior: Transfers the value from the source path to the target path.

Example:

- rename:
    from: [log.user_name, log.login]
    to: origin.user
    where: 'exists("log.user_name")'

3. cast

Converts field types.

• Fields: fields (Array), to (Target type), where (Optional).

• Supported Types: int, float, string, bool, []string.

Example:

- cast:
    fields: [origin.port]
    to: int

4. delete

Deletes fields from the log in order to improve storage and indexing.

• Fields: fields (Array), where (Optional).

Example:

- delete: 
    fields: [log.temporary_header, log.internal_id]

5. grok

Matching patterns in unstructured text.

• Fields: source (Defaults to raw), patterns (List of { fieldName, pattern }), where (Optional).

• Placement: Uses the fieldName provided in the pattern list literally. To use standard namespaces, indicate them (e.g., origin.ip). Per convention, custom fields should use log..

• Extensibility: Users can add or modify standard patterns through the Unified Security Stack WebUI.

Default Standard Patterns

Example:

- grok:
    source: raw
    patterns:
      - fieldName: origin.ip
        pattern: '{{.ipv4}}'
      - fieldName: log.event_id
        pattern: 'ID: {{.integer}}'

6. kv (Key-Value)

Pulls key-value pairs out of a string.

• Fields: source, fieldSplit (Separator between pairs), valueSplit (Separator between key and value), where (Optional).

• Placement: Like the json step, all extracted keys are automatically prefixed with log..

Example:

- kv: 
    source: raw
    fieldSplit: " " 
    valueSplit: "="

7. trim

Sanitizes strings through the removal of prefixes, suffixes, or pattern matches.

• Fields: fields (Array), function (prefix, suffix, substring, regex), substring (the string or pattern to trim), where (Optional).

Example:

- trim: 
    function: suffix
    substring: ".local"
    fields: [origin.host]

8. add

Inserts a constant value into a field.

• Fields: function (string), params (Map), where (Optional).

• Required Params: key (target path), value (the specific value to be added).

• Placement: Uses the key literally.

Example:

- add: 
    function: string
    params: 
      key: log.category
      value: security

9. reformat

Transforms field formats, mainly for timestamps.

• Fields: fields (Array), function (time), fromFormat (Go layout), toFormat (Go layout), where (Optional).

• Behavior: Overwrites the value in the specified fields.

Example:

- reformat:
    fields: [deviceTime]
    function: time
    fromFormat: 'Jan 02 15:04:05'
    toFormat: '2006-01-02T15:04:05Z'

10. csv

Reads comma-separated values.

• Fields: source, separator, headers (Array of target paths), where (Optional).

• Placement: Uses the names provided in headers literally.

Example:

- csv:
    source: raw
    separator: ","
    headers: [log.id, origin.user, action, actionResult]

11. dynamic

Invokes an external gRPC plugin.

• Fields: plugin (Name), params (Key-value map), where (Optional).

Example:

- dynamic:
    plugin: com.synoti.geolocation
    params: 
      source: origin.ip
      destination: origin.geolocation

12. drop

Drops the log right away.

• Required: where (CEL condition).

Example:

- drop:
    where: equals("origin.ip", "127.0.0.1")