DOC-129

docs/includes/activate-audit-log-storage-elasticsearch.md

@@ -0,0 +1,7 @@
+To activate audit log storage in Elasticsearch, update your TheHive configuration file with the following settings:
+
+```json
+audit {
+  storage = elasticsearch
+}
+```

docs/includes/configure-index-ilm-elasticsearch.md

@@ -0,0 +1,113 @@
+Configure the default index template and Index Lifecycle Management (ILM) for `thehive-audits` using the following settings:
+
+* `index-name`
+* `health-request-timeout`
+* `create.ext.number_of_replicas`
+* `create.ext.number_of_shards`
+* `request-timeout`
+* `hot.rollover.maxAge`
+* `hot.rollover.maxSize`
+* `delete.minAge`
+
+#### Index template
+
+The index template ensures consistent storage and indexing of audit logs.
+
+Default index template:
+
+```json
+{
+  "index": {
+    "lifecycle": {
+      "name": "thehive-audits"
+    },
+    "number_of_shards": "1",
+    "number_of_replicas": "1"
+  }
+}
+```
+
+#### ILM
+
+ILM manages the storage, rollover, and deletion of audit logs over time, optimizing long-term storage.
+
+Default ILM:
+
+```json
+{
+  "policy": {
+    "phases": {
+      "hot": {
+        "min_age": "0ms",
+        "actions": {
+          "set_priority": {
+            "priority": 100
+          }
+        }
+      }
+    }
+  }
+}
+```
+
+!!! example "Example"
+    To create a new index every day or when it reaches 1 GB, and retain it for 7 days, use the following configuration:
+
+    - Index template:
+
+    ```json
+    audit {
+        storage = elasticsearch
+        elasticsearch {
+            hot.rollover {
+                maxAge = 1d
+                maxSize = 1g
+            }
+            delete.minAge = 7d
+        }
+    }
+    ```
+
+    - ILM:
+
+    ```json
+    {
+        "policy": {
+            "phases": {
+                "hot": {
+                    "min_age": "0ms",
+                    "actions": {
+                        "set_priority": {
+                            "priority": 100
+                        },
+                        "rollover": {
+                            "max_primary_shard_size": "1gb",
+                            "max_age": "1d"
+                        }
+                    }
+                },
+                "delete": {
+                    "min_age": "7d",
+                    "actions": {
+                        "delete": {
+                            "delete_searchable_snapshot": true
+                        }
+                    }
+                }
+            }
+        }
+    }
+    ```
+
+!!! warning "API calls changes"
+    After migrating to Elasticsearch, the following API calls no longer work with audit logs stored in Elasticsearch:
+
+    - `listAudit`
+    - `listAuditFromObject`
+    - `getAudit`
+
+    Instead, use the following API calls:
+
+    - `list`
+    - `count`
+    - `get`

docs/thehive/installation/step-by-step-installation-guide.md

@@ -363,8 +363,10 @@ For additional configuration options, refer to:
 
 [Elasticsearch](https://www.elastic.co/elasticsearch) is a robust data indexing and search engine. TheHive uses it to manage data indices efficiently.
 
+<!-- md:version 5.5 --> Elasticsearch can also replace Apache Cassandra (JanusGraph) for storing audit logs.
+
 !!! note "Elasticsearch support"
-    Starting from version 5.3, TheHive supports Elasticsearch 8.0 and 7.x. Earlier versions only support Elasticsearch 7.x.
+    Starting from version 5.3, TheHive supports Elasticsearch 8.0 and 7.x. Earlier versions only support Elasticsearch 7.x. If you want to use Elasticsearch to store your audit logs, ensure that you are using Elasticsearch 7.17 or later.
 
 !!! note "OpenSearch support"
     Starting from version 5.3, TheHive supports [OpenSearch](https://opensearch.org/) for advanced use cases.
@@ -466,11 +468,11 @@ You can configure Elasticsearch by modifying settings within the `/etc/elasticse
 
     Set the `path.logs` and `path.data` parameters to the desired directories, such as `/var/log/elasticsearch` and `/var/lib/elasticsearch`, respectively.
 
-**7. Optional: Configure X-Pack security:**
+**7. (Optional) Configure X-Pack security:**
 
     If you're not using X-Pack security, ensure that `xpack.security.enabled` is set to `false`.
 
-**8. Optional: Configure script allowed types:**
+**8. (Optional) Configure script allowed types:**
 
     If needed, set the `script.allowed_types` parameter to specify allowed script types.
 
@@ -514,6 +516,27 @@ You can configure Elasticsearch by modifying settings within the `/etc/elasticse
     * Similar to data and files, include indexes in the backup policy to ensure their preservation.
     * Remove and re-create indexes as needed.
 
+**12. <!-- md:version 5.5 --> (Optional) Configure audit log storage:**
+
+By default, TheHive stores audit logs in Apache Cassandra via JanusGraph. However, if your organization generates a large volume of audit logs, you can switch to Elasticsearch. 
+
+Elasticsearch offers better performance, reduced latency, and advanced search capabilities, making it ideal for managing and retrieving large amounts of audit data.
+
+!!! warning "Audit logs visibility"
+    - With Elasticsearch, audit logs retain the visibility they had at the time of creation, regardless of the current visibility of the case. This means that even if [you set restricted visibility for a case](../user-guides/analyst-corner/cases/restrict-visibility-case.md), audit logs remain visible to all users.
+    - With the default configuration using Apache Cassandra (JanusGraph), audit logs immediately become private if the visibility of the associated case is restricted. This means that even if audit logs were originally public, they will be hidden from all users [once the case visibility is restricted](../user-guides/analyst-corner/cases/restrict-visibility-case.md).
+
+!!! warning "Prerequisites"
+    Regularly [back up your Elasticsearch indices](https://www.elastic.co/docs/deploy-manage/tools/snapshot-and-restore) to ensure you can recover audit logs in the event of an incident. This is critical for maintaining the integrity and availability of your data.
+
+a. Activate audit log storage.
+
+{!includes/activate-audit-log-storage-elasticsearch.md!}
+
+b. Configure index template and Index Lifecycle Management (ILM).
+
+{!includes/configure-index-ilm-elasticsearch.md!}
+
 ### Start the service
 
 === "DEB"

docs/thehive/operations/configure-audit-logs-storage-elasticsearch.md

@@ -0,0 +1,78 @@
+# How to Configure Audit Log Storage in Elasticsearch
+
+<!-- md:version 5.5 -->
+
+You can choose to store audit logs in either Apache Cassandra via JanusGraph or in [Elasticsearch](https://www.elastic.co/enterprise-search) in TheHive.
+
+By default, TheHive stores audit logs in Apache Cassandra via JanusGraph.
+
+This topic provides step-by-step instructions for configuring TheHive's audit log storage in Elasticsearch for existing instances, including the option to migrate historical logs from JanusGraph.
+
+!!! info "Reasons to consider Elasticsearch"
+    Elasticsearch is better suited for managing large volumes of audit logs. It enhances performance by efficiently handling data, reducing latency, and offering advanced search capabilities. If your organization generates a significant amount of audit logs, migrating to Elasticsearch can improve both data management and retrieval.
+
+!!! warning "Audit logs visibility"
+    With Elasticsearch, audit logs retain the visibility they had at the time of creation, regardless of the current visibility of the case. This means that even if [you set restricted visibility for a case](../user-guides/analyst-corner/cases/restrict-visibility-case.md), audit logs remain visible to all users.
+
+!!! warning "No rollback possible"
+    Since TheHive can only use one audit storage system at a time, when you switch to Elasticsearch, the audit logs stored in JanusGraph become unavailable. TheHive doesn't support transferring audit logs back from Elasticsearch to JanusGraph. 
+
+!!! note "New installations"
+    For new installations, refer to the [Step-by-Step Guide](../installation/step-by-step-installation-guide.md) for instructions.
+
+## Prerequisites
+
+### Use Elasticsearch 7.17 or later
+
+Make sure you're using Elasticsearch version 7.17 or later to guarantee compatibility with TheHive's audit log storage.
+
+### Back up Elasticsearch indices
+
+Regularly [back up your Elasticsearch indices](https://www.elastic.co/docs/deploy-manage/tools/snapshot-and-restore) to ensure you can recover audit logs in the event of an incident. This is critical for maintaining the integrity and availability of your data.
+
+## Step 1: Activate audit log storage in Elasticsearch
+
+{!includes/activate-audit-log-storage-elasticsearch.md!}
+
+## Step 2: Configure index template and Index Lifecycle Management (ILM)
+
+{!includes/configure-index-ilm-elasticsearch.md!}
+
+## (Optional) Step 3: Migrate historical audit logs to Elasticsearch
+
+TheHive provides a script to assist with migrating historical audit logs from JanusGraph to Elasticsearch. Run the script only if you need to transfer historical audit logs to Elasticsearch.
+
+!!! warning "JanusGraph audit log deletion"
+    By default, when migrating audit logs from JanusGraph to Elasticsearch, the logs are deleted from JanusGraph. If you wish to keep the audit logs in JanusGraph while migrating to Elasticsearch, you can prevent their deletion by using a specific option.
+
+Run the `migrateAudits` script:
+
+* If you installed TheHive on a server:
+
+  ``` bash
+  ./migrateAudits --audit-from-date <date> -c </etc/thehive/application.conf>
+  ```
+
+* If you deployed TheHive with a Docker image:
+
+  ``` bash
+  docker run --network host --rm -ti -v </path/to/your/application.conf>:/etc/thehive/application.conf:ro -v </path/to/your/logback.xml>:/etc/thehive/logback.xml:ro strangebee/thehive:5.5.0-1-SNAPSHOT  migrateAudits -Dlogback.configurationFile=/etc/thehive/logback.xml -- --audit-from-date <date> -c /etc/thehive/application.conf
+  ```
+
+Available options are:
+
+```
+-v, --version: Displays the version of the migration tool.
+-h, --help: Displays help information.
+-c, --config <file> : Specifies the global configuration file for TheHive.
+-y, --transaction-pagesize <value>: Defines the page size for each transaction during the migration.
+-n, --no-deletion: Prevents the deletion of audit logs from JanusGraph after they are migrated to Elasticsearch.
+--audit-from-date <date>: Migrates only the audit logs created from the specified date. The date format is yyyyMMdd[HH[mm[ss]]] or yyyy/MM/dd HH:mm:ss.
+--audit-until-date <date>: Migrates only the audits created until the specified date. The date format is yyyyMMdd[HH[mm[ss]]] or yyyy/MM/dd HH:mm:ss.
+```
+
+<h2>Next steps</h2>
+
+* [Back up and restore](../operations/backup-restore/overview.md)
+* [Monitoring](monitoring.md)
+* [Troubleshooting](troubleshooting.md)

docs/thehive/user-guides/analyst-corner/cases/about-cases.md

@@ -126,7 +126,9 @@ Unauthorized users can't be assigned to the case.
     A case with restricted visibility still triggers [configured notifications](../../organization/configure-organization/manage-notifications/about-notifications.md), regardless of who can view the case.
 
 !!! warning "Audit logs behavior"
-    When visibility is restricted on a case, all associated [audit logs](../../organization/about-audit-logs.md), including those created earlier, immediately become private.
+    When visibility is restricted on a case:  
+    - If you're using the default configuration with Apache Cassandra (JanusGraph) to store your audit logs, all associated [audit logs](../../organization/about-audit-logs.md), including those created earlier, immediately become private.  
+    - If you're using [Elasticsearch to store your audit logs](../../../operations/configure-audit-logs-storage-elasticsearch.md), all associated [audit logs](../../organization/about-audit-logs.md) remain public. Audit logs retain the visibility they had at the time of creation, regardless of the current visibility of the case.
 
 #### Indicators
 

mkdocs.yml

@@ -390,6 +390,7 @@ nav:
       - 'Cassandra Cluster Operations': thehive/operations/cassandra-cluster.md
       - 'Cassandra Security Operations': thehive/operations/cassandra-security.md
       - 'MinIO Cluster Operations': thehive/operations/minio-cluster.md
+      - 'Configure Audit Log Storage in Elasticsearch': thehive/operations/configure-audit-logs-storage-elasticsearch.md
       - 'Backup & Restore Operations': 
         - 'Overview':  thehive/operations/backup-restore/overview.md
         - 'Backup Process':