Development Workflow

This guide describes the standard workflow for building and deploying new detection logic and parsing filters.

Rule Development Lifecycle

1. Identify the Threat: Investigate the particular attack pattern, for example a MITRE ATT&CK technique.

2. Analyze the Logs: Examine sample logs in the Unified Security Stack Log Explorer to pinpoint unique identifiers for the threat.

3. Draft the YAMLGenerate a new rule file. Specify metadata (id, name, impact).

4. Implement Conditions: Write the where expression and define any required correlation logic.

5. Local ValidationValidate the YAML syntax with a YAML linter.

6. Deployment: Add the rule to the rules directory of the EventProcessor.

7. Simulation: Create attack logs and confirm that an alert is triggered and correctly deduplicated.

Filter Development Lifecycle

1. Source DiscoveryLocate a new log source (for example, a new firewall brand).

2. Schema Mapping: Map the raw log fields to the Standard Event Schema.

3. Pipeline Construction:

   • Start with a json or kv take a step if the log is structured.

• Use grok to handle unstructured data strings.

• Add cast steps for types and rename for standardization.

1. Enrichment: Add dynamic procedures for geolocation or TI lookups.

2. Sanitization: Remove temporary or highly redundant fields in the log.* namespace for storage optimization.

   Note: The raw This field is safeguarded for audit purposes and cannot be removed.

3. Testing: Use the Playground or deploy to a development environment and check Log Explorer to confirm fields display correctly.

Debugging Tips

• Check Logs: Watch the EventProcessor logs for parsing error or rule evaluation failed messages.

• Isolate: If a filter fails, comment out steps one at a time to locate the cause.

• Validate Schema: Ensure that the types you cast to align with the expected types in the rules.

• Audit Trace: Remember that the raw The field always retains the original message, which helps verify your parsing logic.