Performance Optimization

Rule Optimization

• Data Type Narrowing: Rules are assessed only against the data types they are designed for.

• Efficient CEL: Avoid large array lookups or deep nested loops in where clauses.

• Correlation Windows: Use the smallest possible within timeframe that satisfies the logic.

• Limit Correlation: Querying millions of logs is costly. Narrow results as much as possible by using with fields.

Filter Optimization

• Early Exit: Use where statements that bypass needless steps.

• JSON over Regex: If a log is JSON, always use the json step instead of grok. It is considerably faster.

• Field Cleanup: Always delete the raw field. Long raw strings raise indexing latency and disk usage.

System Resources

• CPU: Parsing and CEL evaluation consume significant CPU. Increase cores to reach higher EPS (Events Per Second).

• RAM: The EventProcessor cache and OpenSearch depend on system memory.

• I/O: Store OpenSearch data directories on SSDs to prevent bottlenecks when correlation searches run at high volume.

Troubleshooting Common Issues

SymptomPotential CauseSolutionRule not triggeringMismatched dataType or where logic.Check Log Explorer to verify the event type is correct and every field named in the rule is present.Filter not parsingIncorrect source field or syntax error.Validate that the source field (e.g., raw) is present in the input. Review processor logs for YAML parsing errors.Alert fatigueMissing deduplicateBy.Group alerts by origin.ip or origin.user to consolidate findings.High LatencyComplex grok or large correlation windows.Optimize regex patterns or reduce the within time window.

Debugging Workflow

1. Logs: Check /var/log/synoti/eventprocessor.log.

2. Validator: Run a YAML linter against your rules/filters.

3. Trace: Send one test log with an event generator and follow it across the system.