Alongside the core EventProcessor, several tools are provided to support data ingestion, testing, and debugging.

opensearch-fetcher

The opensearch-fetcher offers a way to extract logs from an OpenSearch index and save them as individual JSON files. This is very useful for collecting real-world test data for the Playground.

Usage

./opensearch-fetcher -url "http://localhost:9200" -index "v1-7-22-log-*" -size 50 -output "./test_data"

Options

FlagDescriptionDefault-urlOpenSearch server URLhttp://192.168.1.18:9200-index(Required) Index or pattern to fetch from-userUsername for authentication-passwordPassword for authentication-outputDirectory to save JSON filesfetched_logs-sizeNumber of logs to fetch100-querySearch query (Query String syntax)*-startStart date (RFC3339)-endEnd date (RFC3339)Current time

log-pusher

The log-pusher serves to send JSON log files to a running EventProcessor instance via gRPC. It emulates an input plugin and is helpful for replaying logs or conducting load tests.

Usage

./log-pusher -input "./test_data" -socket "/workdir/sockets/engine_server.sock"

Options

FlagDescriptionDefault-input(Required) Directory containing JSON log files-socketPath to the engine_server.sock$WORK_DIR/sockets/engine_server.sock

The log-pusher will recursively walk the input directory, read all .json files, then streams them to the processor. It waits for acknowledgments (ACKs) from the server to confirm data delivery.