synoti logosynoti logo
Configuration & Implementation Data retention

Data Retention

Defines policies for storing and managing log data in Unified Security Stack, ensuring compliance with security regulations and optimizing storage efficiency

Image

1. Index Rollover

With the Index Rollover capability in Unified Security Stack, data is managed efficiently because indices are deleted automatically after they satisfy certain criteria, for example data size or age.

How to Configure Index Rollover:

  1. Go to Application Management > Data Retention.

  2. Specify the retention period you want, measured in days.

  3. To confirm your changes, press Save.

Key Features:

  • Delete Days: This lets you set the duration for data retention. Once that period ends, the data is marked for deletion, which can assist with freeing space and preserving system performance.

Note:* Note: The system automatically takes action when the disk usage reaches*
85%, so system performance remains unaffected.

2. Snapshot Archiving

Understanding Hot and Cold Storage:

  • Hot Storage: Actively accessed data is stored here. It is built for real-time access, which guarantees fast operations.

  • Cold Storage: This kind of storage is intended for archived data. Although it is not accessed often, such data is essential for audits, compliance, or retrospective analysis.

Snapshot Archiving is a vital capability for protecting data across long periods. Even when indices are deleted according to retention settings, snapshots make sure a record of that data remains.

Configuring Snapshot Path in Linux:

Snapshots are automatically saved to /synoti/opensearch/backups. To point these snapshots toward a different drive or remote location:

  1. Mount the Remote Drive or Different Drive:

sudo mount -t nfs REMOTE-SERVER:/remote/path /synoti/opensearch/backups

Ensure to replace REMOTE-SERVER:/remote/path using your actual server and path information.

  1. Add to /etc/fstab for Persistence:
    Add the mount details to /etc/fstab so that the remote drive stays mounted after system reboots.

  2. Test the Configuration
    Verify that the mount operates as intended.

sudo mount -a

3. Data Restoration with Cerebro:

For Cerebro to communicate with OpenSearch during the restoration process, the port must be opened. Use the commands below to open and close the port securely:

Open port 9200:

docker service update --publish-add 9200:9200 synoti_node1

Once the restoration is complete, immediately close the port to minimize security risks:

docker service update --publish-rm 9200:9200 synoti_node1

⚠️ Warning: An open port can expose your OpenSearch data to potential risks. Ensure
that you close the port once the restoration process is complete.

Although Cerebro is not part of Unified Security Stack, it is essential for snapshot management and restoration, making it possible to move data from cold to hot storage.

Working with Cerebro:

  1. Installation & Access:

    • Use the official Cerebro installation guide.

  • Once installed, access its interface, typically found at http://localhost:9000.

  1. Restoration Process:

    Link Cerebro with your opensearch node for seamless data restoration

Image


In the more dropdown menu, select "snapshot"
In the snapshot view, choose a repository to restore the backup.