From 0b6a578a965891d07e53554d084989621d9c160c Mon Sep 17 00:00:00 2001 From: AnneLaure1307 Date: Thu, 17 Apr 2025 12:27:21 +0200 Subject: [PATCH 1/9] Added migration page for audit logs --- ...igrate-audit-logs-storage-elasticsearch.md | 51 +++++++++++++++++++ 1 file changed, 51 insertions(+) create mode 100644 docs/thehive/operations/migrate-audit-logs-storage-elasticsearch.md diff --git a/docs/thehive/operations/migrate-audit-logs-storage-elasticsearch.md b/docs/thehive/operations/migrate-audit-logs-storage-elasticsearch.md new file mode 100644 index 0000000000..c9f687e4fc --- /dev/null +++ b/docs/thehive/operations/migrate-audit-logs-storage-elasticsearch.md @@ -0,0 +1,51 @@ +# How to Migrate Audit Log Storage to Elasticsearch + +[Badge version 5.5] + +TheHive allows you to choose between storing audit logs in Apache Cassandra via JanusGraph or in [Elasticsearch](https://www.elastic.co/enterprise-search). + +This topic provides step-by-step instructions for migrating TheHive's audit log storage to Elasticsearch. + +!!! info "Reasons to consider Elasticsearch" + Elasticsearch is better suited for managing large volumes of audit logs. It enhances performance by efficiently handling data, reducing latency, and offering advanced search capabilities. If your organization generates a significant amount of audit logs, migrating to Elasticsearch can improve both data management and accessibility. + +!!! warning "No rollback possible" + TheHive doesn't support transferring audit logs back from Elasticsearch to JanusGraph. Once migrated, the audit logs in Elasticsearch can't be rolled back. + +!!! note "New installations" + For new installations, refer to the [Step-by-Step Guide](../installation/step-by-step-installation-guide.md) for instructions. + +Two options are possible depending on your organization's requirements and needs: + +* [Migrate without historical audit logs](#migrate-without-historical-audit-logs) +* [Migrate with historical audit logs](#migrate-with-historical-audit-logs) + +## Prerequisites + +### Use Elasticsearch 7.17 or later + +Make sure you're using Elasticsearch version 7.17 or later to guarantee compatibility with TheHive's audit log storage. + +### Back up Elasticsearch indices + +Regularly [back up your Elasticsearch indices](https://www.elastic.co/docs/deploy-manage/tools/snapshot-and-restore) to ensure that audit logs can be recovered in case of an incident. This is critical for maintaining the integrity and availability of your data. + +### Activate audit log storage in Elasticsearch + +To activate audit log storage in Elasticsearch, update your TheHive configuration file with the following settings: + +```bash +audit { + storage = elasticsearch +} +``` +By default, the index is created as follows: + +* Index template: `thehive-audits` +* Index lifecycle policy (ILM): `thehive-audits` (the datastream, template, and ILM always share the same name) + +## Migrate without historical audit logs + +## Migrate with historical audit logs + +

Next steps

\ No newline at end of file From 01c02f70419a166e0244f13526b92750daecc4b7 Mon Sep 17 00:00:00 2001 From: AnneLaure1307 Date: Thu, 17 Apr 2025 15:44:43 +0200 Subject: [PATCH 2/9] Completed the doc --- .../api-calls-elasticsearch-audit-logs.md | 11 ++ ...igrate-audit-logs-storage-elasticsearch.md | 119 ++++++++++++++++-- 2 files changed, 122 insertions(+), 8 deletions(-) create mode 100644 docs/includes/api-calls-elasticsearch-audit-logs.md diff --git a/docs/includes/api-calls-elasticsearch-audit-logs.md b/docs/includes/api-calls-elasticsearch-audit-logs.md new file mode 100644 index 0000000000..03759d7d02 --- /dev/null +++ b/docs/includes/api-calls-elasticsearch-audit-logs.md @@ -0,0 +1,11 @@ +!!! warning "API calls changes" + After migrating to Elasticsearch, the following API calls no longer work with audit logs stored in Elasticsearch: + + - `listAudit` + - `listAuditFromObject` + + Instead, use the following API calls: + + - `list` + - `count` + - `get` \ No newline at end of file diff --git a/docs/thehive/operations/migrate-audit-logs-storage-elasticsearch.md b/docs/thehive/operations/migrate-audit-logs-storage-elasticsearch.md index c9f687e4fc..d50a7f0d3d 100644 --- a/docs/thehive/operations/migrate-audit-logs-storage-elasticsearch.md +++ b/docs/thehive/operations/migrate-audit-logs-storage-elasticsearch.md @@ -7,10 +7,10 @@ TheHive allows you to choose between storing audit logs in Apache Cassandra via This topic provides step-by-step instructions for migrating TheHive's audit log storage to Elasticsearch. !!! info "Reasons to consider Elasticsearch" - Elasticsearch is better suited for managing large volumes of audit logs. It enhances performance by efficiently handling data, reducing latency, and offering advanced search capabilities. If your organization generates a significant amount of audit logs, migrating to Elasticsearch can improve both data management and accessibility. + Elasticsearch is better suited for managing large volumes of audit logs. It enhances performance by efficiently handling data, reducing latency, and offering advanced search capabilities. If your organization generates a significant amount of audit logs, migrating to Elasticsearch can improve both data management and retrieval. !!! warning "No rollback possible" - TheHive doesn't support transferring audit logs back from Elasticsearch to JanusGraph. Once migrated, the audit logs in Elasticsearch can't be rolled back. + TheHive doesn't support transferring audit logs back from Elasticsearch to JanusGraph. Once migrated, the audit logs in Elasticsearch can't be rolled back. Since TheHive can only use one audit storage system at a time, when you switch to Elasticsearch, the audit logs stored in JanusGraph become unavailable. !!! note "New installations" For new installations, refer to the [Step-by-Step Guide](../installation/step-by-step-installation-guide.md) for instructions. @@ -22,15 +22,15 @@ Two options are possible depending on your organization's requirements and needs ## Prerequisites -### Use Elasticsearch 7.17 or later +### Step 1: Use Elasticsearch 7.17 or later Make sure you're using Elasticsearch version 7.17 or later to guarantee compatibility with TheHive's audit log storage. -### Back up Elasticsearch indices +### Step 2: Back up Elasticsearch indices Regularly [back up your Elasticsearch indices](https://www.elastic.co/docs/deploy-manage/tools/snapshot-and-restore) to ensure that audit logs can be recovered in case of an incident. This is critical for maintaining the integrity and availability of your data. -### Activate audit log storage in Elasticsearch +### Step 3: Activate audit log storage in Elasticsearch To activate audit log storage in Elasticsearch, update your TheHive configuration file with the following settings: @@ -39,13 +39,116 @@ audit { storage = elasticsearch } ``` -By default, the index is created as follows: -* Index template: `thehive-audits` -* Index lifecycle policy (ILM): `thehive-audits` (the datastream, template, and ILM always share the same name) +### Step 4: Configure index template and Index Lifecycle Management (ILM) + +Configure the default index template and Index Lifecycle Management (ILM) for `thehive-audits` using the following settings: + +* `index-name` +* `health-request-timeout` +* `create.ext.number_of_replicas` +* `create.ext.number_of_shards` +* `request-timeout` +* `hot.rollover.maxAge` +* `hot.rollover.maxSize` +* `delete.minAge` + +#### Index template + +The index template ensures consistent storage and indexing of audit logs. + +Default index template: + +```bash +{ + "index": { + "lifecycle": { + "name": "thehive-audits" + }, + "number_of_shards": "1", + "number_of_replicas": "1" + } +} +``` + +#### ILM + +ILM manages the storage, rollover, and deletion of audit logs over time, optimizing long-term storage. + +Default ILM: + +```bash +{ + "policy": { + "phases": { + "hot": { + "min_age": "0ms", + "actions": { + "set_priority": { + "priority": 100 + } + } + } + } + } +} +``` + +!!! example "Configure rollover and retention" + To create a new index every day or when it reaches 1 GB, and retain it for 7 days, use the following configuration: + + * Index template: + + ```bash + audit { + storage = elasticsearch + elasticsearch { + hot.rollover { + maxAge = 1d + maxSize = 1g + } + delete.minAge = 7d + } + } + ``` + + * ILM: + + ```bash + { + "policy": { + "phases": { + "hot": { + "min_age": "0ms", + "actions": { + "set_priority": { + "priority": 100 + }, + "rollover": { + "max_primary_shard_size": "1gb", + "max_age": "1d" + } + } + }, + "delete": { + "min_age": "7d", + "actions": { + "delete": { + "delete_searchable_snapshot": true + } + } + } + } + } + } + ``` ## Migrate without historical audit logs +{!includes/api-calls-elasticsearch-audit-logs.md!} + ## Migrate with historical audit logs +{!includes/api-calls-elasticsearch-audit-logs.md!} +

Next steps

\ No newline at end of file From 2172293572734777e7e4f86176f3885b52985607 Mon Sep 17 00:00:00 2001 From: AnneLaure1307 Date: Thu, 17 Apr 2025 16:45:16 +0200 Subject: [PATCH 3/9] Finalized doc --- .../api-calls-elasticsearch-audit-logs.md | 1 + ...igure-audit-logs-storage-elasticsearch.md} | 54 +++++++++++++------ 2 files changed, 39 insertions(+), 16 deletions(-) rename docs/thehive/operations/{migrate-audit-logs-storage-elasticsearch.md => configure-audit-logs-storage-elasticsearch.md} (59%) diff --git a/docs/includes/api-calls-elasticsearch-audit-logs.md b/docs/includes/api-calls-elasticsearch-audit-logs.md index 03759d7d02..bc30eb464f 100644 --- a/docs/includes/api-calls-elasticsearch-audit-logs.md +++ b/docs/includes/api-calls-elasticsearch-audit-logs.md @@ -3,6 +3,7 @@ - `listAudit` - `listAuditFromObject` + - `getAudit` Instead, use the following API calls: diff --git a/docs/thehive/operations/migrate-audit-logs-storage-elasticsearch.md b/docs/thehive/operations/configure-audit-logs-storage-elasticsearch.md similarity index 59% rename from docs/thehive/operations/migrate-audit-logs-storage-elasticsearch.md rename to docs/thehive/operations/configure-audit-logs-storage-elasticsearch.md index d50a7f0d3d..fd916aed25 100644 --- a/docs/thehive/operations/migrate-audit-logs-storage-elasticsearch.md +++ b/docs/thehive/operations/configure-audit-logs-storage-elasticsearch.md @@ -1,36 +1,31 @@ -# How to Migrate Audit Log Storage to Elasticsearch +# How to Configure Audit Log Storage in Elasticsearch [Badge version 5.5] TheHive allows you to choose between storing audit logs in Apache Cassandra via JanusGraph or in [Elasticsearch](https://www.elastic.co/enterprise-search). -This topic provides step-by-step instructions for migrating TheHive's audit log storage to Elasticsearch. +This topic provides step-by-step instructions for configuring TheHive's audit log storage in Elasticsearch, including the option to migrate historical logs from JanusGraph. !!! info "Reasons to consider Elasticsearch" Elasticsearch is better suited for managing large volumes of audit logs. It enhances performance by efficiently handling data, reducing latency, and offering advanced search capabilities. If your organization generates a significant amount of audit logs, migrating to Elasticsearch can improve both data management and retrieval. !!! warning "No rollback possible" - TheHive doesn't support transferring audit logs back from Elasticsearch to JanusGraph. Once migrated, the audit logs in Elasticsearch can't be rolled back. Since TheHive can only use one audit storage system at a time, when you switch to Elasticsearch, the audit logs stored in JanusGraph become unavailable. + TheHive doesn't support transferring audit logs back from Elasticsearch to JanusGraph. Since TheHive can only use one audit storage system at a time, when you switch to Elasticsearch, the audit logs stored in JanusGraph become unavailable. !!! note "New installations" For new installations, refer to the [Step-by-Step Guide](../installation/step-by-step-installation-guide.md) for instructions. -Two options are possible depending on your organization's requirements and needs: - -* [Migrate without historical audit logs](#migrate-without-historical-audit-logs) -* [Migrate with historical audit logs](#migrate-with-historical-audit-logs) - ## Prerequisites -### Step 1: Use Elasticsearch 7.17 or later +### Use Elasticsearch 7.17 or later Make sure you're using Elasticsearch version 7.17 or later to guarantee compatibility with TheHive's audit log storage. -### Step 2: Back up Elasticsearch indices +### Back up Elasticsearch indices Regularly [back up your Elasticsearch indices](https://www.elastic.co/docs/deploy-manage/tools/snapshot-and-restore) to ensure that audit logs can be recovered in case of an incident. This is critical for maintaining the integrity and availability of your data. -### Step 3: Activate audit log storage in Elasticsearch +## Step 1: Activate audit log storage in Elasticsearch To activate audit log storage in Elasticsearch, update your TheHive configuration file with the following settings: @@ -40,7 +35,7 @@ audit { } ``` -### Step 4: Configure index template and Index Lifecycle Management (ILM) +## Step 2: Configure index template and Index Lifecycle Management (ILM) Configure the default index template and Index Lifecycle Management (ILM) for `thehive-audits` using the following settings: @@ -143,12 +138,39 @@ Default ILM: } ``` -## Migrate without historical audit logs - {!includes/api-calls-elasticsearch-audit-logs.md!} -## Migrate with historical audit logs +## (Optional) Step 3: Migrate historical audit logs to Elasticsearch -{!includes/api-calls-elasticsearch-audit-logs.md!} +TheHive provides a tool to help migrate historical audit logs from JanusGraph to Elasticsearch. Proceed with these instructions only if you need to transfer historical audit logs to Elasticsearch. + +!!! warning "JanusGraph audit log deletion" + By default, when migrating audit logs from JanusGraph to Elasticsearch, the logs are deleted from JanusGraph. If you wish to keep the audit logs in JanusGraph while migrating to Elasticsearch, you can prevent their deletion by using a specific option. + +Run the migrateAudits tool: + +* If TheHive is manually installed on a server: + +``` bash +./migrateAudits --audit-from-date -c +``` + +* If TheHive is deployed with a Docker image: + +``` bash +docker run --network host --rm -ti -v :/etc/thehive/application.conf:ro -v :/etc/thehive/logback.xml:ro strangebee/thehive:5.5.0-1-SNAPSHOT migrateAudits -Dlogback.configurationFile=/etc/thehive/logback.xml -- --audit-from-date -c /etc/thehive/application.conf +``` + +Available options are: + +``` +-v, --version: Displays the version of the migration tool. +-h, --help: Displays help information. +-c, --config : Specifies the global configuration file for TheHive. +-y, --transaction-pagesize : Defines the page size for each transaction during the migration. +-n, --no-deletion: Prevents the deletion of audit logs from JanusGraph after they are migrated to Elasticsearch. +--audit-from-date : Migrates only the audit logs created from the specified date. The date format is yyyyMMdd[HH[mm[ss]]] or yyyy/MM/dd HH:mm:ss. +--audit-until-date : Migrates only the audits created until the specified date. The date format is yyyyMMdd[HH[mm[ss]]] or yyyy/MM/dd HH:mm:ss. +```

Next steps

\ No newline at end of file From abc8cb8fb5f33e3c44703aac164ba1237d645683 Mon Sep 17 00:00:00 2001 From: AnneLaure1307 Date: Tue, 22 Apr 2025 10:59:00 +0200 Subject: [PATCH 4/9] Modified stepbystep guide --- ...ctivate-audit-log-storage-elasticsearch.md | 7 + .../api-calls-elasticsearch-audit-logs.md | 12 -- .../configure-index-ilm-elasticsearch.md | 113 ++++++++++++++++ .../step-by-step-installation-guide.md | 19 ++- ...figure-audit-logs-storage-elasticsearch.md | 121 ++---------------- 5 files changed, 147 insertions(+), 125 deletions(-) create mode 100644 docs/includes/activate-audit-log-storage-elasticsearch.md delete mode 100644 docs/includes/api-calls-elasticsearch-audit-logs.md create mode 100644 docs/includes/configure-index-ilm-elasticsearch.md diff --git a/docs/includes/activate-audit-log-storage-elasticsearch.md b/docs/includes/activate-audit-log-storage-elasticsearch.md new file mode 100644 index 0000000000..16bf71934f --- /dev/null +++ b/docs/includes/activate-audit-log-storage-elasticsearch.md @@ -0,0 +1,7 @@ +To activate audit log storage in Elasticsearch, update your TheHive configuration file with the following settings: + +```bash +audit { + storage = elasticsearch +} +``` \ No newline at end of file diff --git a/docs/includes/api-calls-elasticsearch-audit-logs.md b/docs/includes/api-calls-elasticsearch-audit-logs.md deleted file mode 100644 index bc30eb464f..0000000000 --- a/docs/includes/api-calls-elasticsearch-audit-logs.md +++ /dev/null @@ -1,12 +0,0 @@ -!!! warning "API calls changes" - After migrating to Elasticsearch, the following API calls no longer work with audit logs stored in Elasticsearch: - - - `listAudit` - - `listAuditFromObject` - - `getAudit` - - Instead, use the following API calls: - - - `list` - - `count` - - `get` \ No newline at end of file diff --git a/docs/includes/configure-index-ilm-elasticsearch.md b/docs/includes/configure-index-ilm-elasticsearch.md new file mode 100644 index 0000000000..3010187bb6 --- /dev/null +++ b/docs/includes/configure-index-ilm-elasticsearch.md @@ -0,0 +1,113 @@ +Configure the default index template and Index Lifecycle Management (ILM) for `thehive-audits` using the following settings: + +* `index-name` +* `health-request-timeout` +* `create.ext.number_of_replicas` +* `create.ext.number_of_shards` +* `request-timeout` +* `hot.rollover.maxAge` +* `hot.rollover.maxSize` +* `delete.minAge` + +### Index template + +The index template ensures consistent storage and indexing of audit logs. + +Default index template: + +```bash +{ + "index": { + "lifecycle": { + "name": "thehive-audits" + }, + "number_of_shards": "1", + "number_of_replicas": "1" + } +} +``` + +### ILM + +ILM manages the storage, rollover, and deletion of audit logs over time, optimizing long-term storage. + +Default ILM: + +```bash +{ + "policy": { + "phases": { + "hot": { + "min_age": "0ms", + "actions": { + "set_priority": { + "priority": 100 + } + } + } + } + } +} +``` + +!!! example "Configure rollover and retention" + To create a new index every day or when it reaches 1 GB, and retain it for 7 days, use the following configuration: + + - Index template: + + ```bash + audit { + storage = elasticsearch + elasticsearch { + hot.rollover { + maxAge = 1d + maxSize = 1g + } + delete.minAge = 7d + } + } + ``` + + - ILM: + + ```bash + { + "policy": { + "phases": { + "hot": { + "min_age": "0ms", + "actions": { + "set_priority": { + "priority": 100 + }, + "rollover": { + "max_primary_shard_size": "1gb", + "max_age": "1d" + } + } + }, + "delete": { + "min_age": "7d", + "actions": { + "delete": { + "delete_searchable_snapshot": true + } + } + } + } + } + } + ``` + +!!! warning "API calls changes" + After migrating to Elasticsearch, the following API calls no longer work with audit logs stored in Elasticsearch: + + - `listAudit` + - `listAuditFromObject` + - `getAudit` + + Instead, use the following API calls: + + - `list` + - `count` + - `get` \ No newline at end of file diff --git a/docs/thehive/installation/step-by-step-installation-guide.md b/docs/thehive/installation/step-by-step-installation-guide.md index 6a368e232c..fad8a700c0 100644 --- a/docs/thehive/installation/step-by-step-installation-guide.md +++ b/docs/thehive/installation/step-by-step-installation-guide.md @@ -362,8 +362,10 @@ For additional configuration options, refer to: [Elasticsearch](https://www.elastic.co/elasticsearch) is a robust data indexing and search engine. TheHive uses it to manage data indices efficiently. +[Badge version 5.5] Elasticsearch can also replace Apache Cassandra and JanusGraph for storing audit logs. + !!! note "Elasticsearch support" - Starting from version 5.3, TheHive supports Elasticsearch 8.0 and 7.x. Earlier versions only support Elasticsearch 7.x. + Starting from version 5.3, TheHive supports Elasticsearch 8.0 and 7.x. Earlier versions only support Elasticsearch 7.x. If you want to use Elasticsearch to store your audit logs, ensure that you are using Elasticsearch 7.17 or later. !!! note "OpenSearch support" Starting from version 5.3, TheHive supports [OpenSearch](https://opensearch.org/) for advanced use cases. @@ -512,6 +514,21 @@ You can configure Elasticsearch by modifying settings within the `/etc/elasticse - Similar to data and files, include indexes in the backup policy to ensure their preservation. - Remove and re-create indexes as needed. +**12. [Badge version 5.5] Optional: Configure audit log storage:** + +By default, TheHive stores audit logs in Apache Cassandra via JanusGraph. However, if your organization generates a large volume of audit logs, you can switch to Elasticsearch. Elasticsearch offers better performance, reduced latency, and advanced search capabilities, making it ideal for managing and retrieving large amounts of audit data. + +!!! warning "Prerequisites" + Regularly [back up your Elasticsearch indices](https://www.elastic.co/docs/deploy-manage/tools/snapshot-and-restore) to ensure that audit logs can be recovered in case of an incident. This is critical for maintaining the integrity and availability of your data. + +1. Activate audit log storage + + {!includes/activate-audit-log-storage-elasticsearch.md!} + +2. Configure index template and Index Lifecycle Management (ILM) + + {!includes/configure-index-ilm-elasticsearch.md!} + ### Start the service === "DEB" diff --git a/docs/thehive/operations/configure-audit-logs-storage-elasticsearch.md b/docs/thehive/operations/configure-audit-logs-storage-elasticsearch.md index fd916aed25..6aa3d3f4ed 100644 --- a/docs/thehive/operations/configure-audit-logs-storage-elasticsearch.md +++ b/docs/thehive/operations/configure-audit-logs-storage-elasticsearch.md @@ -10,7 +10,7 @@ This topic provides step-by-step instructions for configuring TheHive's audit lo Elasticsearch is better suited for managing large volumes of audit logs. It enhances performance by efficiently handling data, reducing latency, and offering advanced search capabilities. If your organization generates a significant amount of audit logs, migrating to Elasticsearch can improve both data management and retrieval. !!! warning "No rollback possible" - TheHive doesn't support transferring audit logs back from Elasticsearch to JanusGraph. Since TheHive can only use one audit storage system at a time, when you switch to Elasticsearch, the audit logs stored in JanusGraph become unavailable. + Since TheHive can only use one audit storage system at a time, when you switch to Elasticsearch, the audit logs stored in JanusGraph become unavailable. TheHive doesn't support transferring audit logs back from Elasticsearch to JanusGraph. !!! note "New installations" For new installations, refer to the [Step-by-Step Guide](../installation/step-by-step-installation-guide.md) for instructions. @@ -27,122 +27,15 @@ Regularly [back up your Elasticsearch indices](https://www.elastic.co/docs/deplo ## Step 1: Activate audit log storage in Elasticsearch -To activate audit log storage in Elasticsearch, update your TheHive configuration file with the following settings: - -```bash -audit { - storage = elasticsearch -} -``` +{!includes/activate-audit-log-storage-elasticsearch.md!} ## Step 2: Configure index template and Index Lifecycle Management (ILM) -Configure the default index template and Index Lifecycle Management (ILM) for `thehive-audits` using the following settings: - -* `index-name` -* `health-request-timeout` -* `create.ext.number_of_replicas` -* `create.ext.number_of_shards` -* `request-timeout` -* `hot.rollover.maxAge` -* `hot.rollover.maxSize` -* `delete.minAge` - -#### Index template - -The index template ensures consistent storage and indexing of audit logs. - -Default index template: - -```bash -{ - "index": { - "lifecycle": { - "name": "thehive-audits" - }, - "number_of_shards": "1", - "number_of_replicas": "1" - } -} -``` - -#### ILM - -ILM manages the storage, rollover, and deletion of audit logs over time, optimizing long-term storage. - -Default ILM: - -```bash -{ - "policy": { - "phases": { - "hot": { - "min_age": "0ms", - "actions": { - "set_priority": { - "priority": 100 - } - } - } - } - } -} -``` - -!!! example "Configure rollover and retention" - To create a new index every day or when it reaches 1 GB, and retain it for 7 days, use the following configuration: - - * Index template: - - ```bash - audit { - storage = elasticsearch - elasticsearch { - hot.rollover { - maxAge = 1d - maxSize = 1g - } - delete.minAge = 7d - } - } - ``` - - * ILM: - - ```bash - { - "policy": { - "phases": { - "hot": { - "min_age": "0ms", - "actions": { - "set_priority": { - "priority": 100 - }, - "rollover": { - "max_primary_shard_size": "1gb", - "max_age": "1d" - } - } - }, - "delete": { - "min_age": "7d", - "actions": { - "delete": { - "delete_searchable_snapshot": true - } - } - } - } - } - } - ``` - -{!includes/api-calls-elasticsearch-audit-logs.md!} +{!includes/configure-index-ilm-elasticsearch.md!} ## (Optional) Step 3: Migrate historical audit logs to Elasticsearch -TheHive provides a tool to help migrate historical audit logs from JanusGraph to Elasticsearch. Proceed with these instructions only if you need to transfer historical audit logs to Elasticsearch. +TheHive provides a script to assist with migrating historical audit logs from JanusGraph to Elasticsearch. Run the script only if you need to transfer historical audit logs to Elasticsearch. !!! warning "JanusGraph audit log deletion" By default, when migrating audit logs from JanusGraph to Elasticsearch, the logs are deleted from JanusGraph. If you wish to keep the audit logs in JanusGraph while migrating to Elasticsearch, you can prevent their deletion by using a specific option. @@ -173,4 +66,8 @@ Available options are: --audit-until-date : Migrates only the audits created until the specified date. The date format is yyyyMMdd[HH[mm[ss]]] or yyyy/MM/dd HH:mm:ss. ``` -

Next steps

\ No newline at end of file +

Next steps

+ +* [Back up and restore](../operations/backup-restore/overview.md) +* [Monitoring](monitoring.md) +* [Troubleshooting](troubleshooting.md) \ No newline at end of file From c6719369f14dd9a3909d243ef15e0b5593ac59fd Mon Sep 17 00:00:00 2001 From: AnneLaure1307 Date: Tue, 22 Apr 2025 11:11:26 +0200 Subject: [PATCH 5/9] Vale checks --- .../installation/step-by-step-installation-guide.md | 2 +- .../configure-audit-logs-storage-elasticsearch.md | 12 ++++++------ 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/thehive/installation/step-by-step-installation-guide.md b/docs/thehive/installation/step-by-step-installation-guide.md index fad8a700c0..9964014c38 100644 --- a/docs/thehive/installation/step-by-step-installation-guide.md +++ b/docs/thehive/installation/step-by-step-installation-guide.md @@ -519,7 +519,7 @@ You can configure Elasticsearch by modifying settings within the `/etc/elasticse By default, TheHive stores audit logs in Apache Cassandra via JanusGraph. However, if your organization generates a large volume of audit logs, you can switch to Elasticsearch. Elasticsearch offers better performance, reduced latency, and advanced search capabilities, making it ideal for managing and retrieving large amounts of audit data. !!! warning "Prerequisites" - Regularly [back up your Elasticsearch indices](https://www.elastic.co/docs/deploy-manage/tools/snapshot-and-restore) to ensure that audit logs can be recovered in case of an incident. This is critical for maintaining the integrity and availability of your data. + Regularly [back up your Elasticsearch indices](https://www.elastic.co/docs/deploy-manage/tools/snapshot-and-restore) to ensure you can recover audit logs in the event of an incident. This is critical for maintaining the integrity and availability of your data. 1. Activate audit log storage diff --git a/docs/thehive/operations/configure-audit-logs-storage-elasticsearch.md b/docs/thehive/operations/configure-audit-logs-storage-elasticsearch.md index 6aa3d3f4ed..248094e77d 100644 --- a/docs/thehive/operations/configure-audit-logs-storage-elasticsearch.md +++ b/docs/thehive/operations/configure-audit-logs-storage-elasticsearch.md @@ -2,9 +2,9 @@ [Badge version 5.5] -TheHive allows you to choose between storing audit logs in Apache Cassandra via JanusGraph or in [Elasticsearch](https://www.elastic.co/enterprise-search). +You can choose to store audit logs in either Apache Cassandra via JanusGraph or in [Elasticsearch](https://www.elastic.co/enterprise-search) in TheHive. -This topic provides step-by-step instructions for configuring TheHive's audit log storage in Elasticsearch, including the option to migrate historical logs from JanusGraph. +This topic provides step-by-step instructions for configuring TheHive's audit log storage in Elasticsearch for existing instances, including the option to migrate historical logs from JanusGraph. !!! info "Reasons to consider Elasticsearch" Elasticsearch is better suited for managing large volumes of audit logs. It enhances performance by efficiently handling data, reducing latency, and offering advanced search capabilities. If your organization generates a significant amount of audit logs, migrating to Elasticsearch can improve both data management and retrieval. @@ -23,7 +23,7 @@ Make sure you're using Elasticsearch version 7.17 or later to guarantee compatib ### Back up Elasticsearch indices -Regularly [back up your Elasticsearch indices](https://www.elastic.co/docs/deploy-manage/tools/snapshot-and-restore) to ensure that audit logs can be recovered in case of an incident. This is critical for maintaining the integrity and availability of your data. +Regularly [back up your Elasticsearch indices](https://www.elastic.co/docs/deploy-manage/tools/snapshot-and-restore) to ensure you can recover audit logs in the event of an incident. This is critical for maintaining the integrity and availability of your data. ## Step 1: Activate audit log storage in Elasticsearch @@ -40,15 +40,15 @@ TheHive provides a script to assist with migrating historical audit logs from Ja !!! warning "JanusGraph audit log deletion" By default, when migrating audit logs from JanusGraph to Elasticsearch, the logs are deleted from JanusGraph. If you wish to keep the audit logs in JanusGraph while migrating to Elasticsearch, you can prevent their deletion by using a specific option. -Run the migrateAudits tool: +Run the `migrateAudits` script: -* If TheHive is manually installed on a server: +* If you installed TheHive on a server: ``` bash ./migrateAudits --audit-from-date -c ``` -* If TheHive is deployed with a Docker image: +* If you deployed TheHive with a Docker image: ``` bash docker run --network host --rm -ti -v :/etc/thehive/application.conf:ro -v :/etc/thehive/logback.xml:ro strangebee/thehive:5.5.0-1-SNAPSHOT migrateAudits -Dlogback.configurationFile=/etc/thehive/logback.xml -- --audit-from-date -c /etc/thehive/application.conf From d1629d5f5a8b1f177f8f7a3314ca12bba45b05fa Mon Sep 17 00:00:00 2001 From: AnneLaure1307 Date: Tue, 22 Apr 2025 11:24:49 +0200 Subject: [PATCH 6/9] Rendering checks --- docs/includes/configure-index-ilm-elasticsearch.md | 6 +++--- .../step-by-step-installation-guide.md | 10 +++++----- .../configure-audit-logs-storage-elasticsearch.md | 14 ++++++++------ mkdocs.yml | 1 + 4 files changed, 17 insertions(+), 14 deletions(-) diff --git a/docs/includes/configure-index-ilm-elasticsearch.md b/docs/includes/configure-index-ilm-elasticsearch.md index 3010187bb6..9eb1c95a2d 100644 --- a/docs/includes/configure-index-ilm-elasticsearch.md +++ b/docs/includes/configure-index-ilm-elasticsearch.md @@ -9,7 +9,7 @@ Configure the default index template and Index Lifecycle Management (ILM) for `t * `hot.rollover.maxSize` * `delete.minAge` -### Index template +#### Index template The index template ensures consistent storage and indexing of audit logs. @@ -27,7 +27,7 @@ Default index template: } ``` -### ILM +#### ILM ILM manages the storage, rollover, and deletion of audit logs over time, optimizing long-term storage. @@ -50,7 +50,7 @@ Default ILM: } ``` -!!! example "Configure rollover and retention" +!!! example "Example" To create a new index every day or when it reaches 1 GB, and retain it for 7 days, use the following configuration: - Index template: diff --git a/docs/thehive/installation/step-by-step-installation-guide.md b/docs/thehive/installation/step-by-step-installation-guide.md index 9964014c38..db74c5a09f 100644 --- a/docs/thehive/installation/step-by-step-installation-guide.md +++ b/docs/thehive/installation/step-by-step-installation-guide.md @@ -362,7 +362,7 @@ For additional configuration options, refer to: [Elasticsearch](https://www.elastic.co/elasticsearch) is a robust data indexing and search engine. TheHive uses it to manage data indices efficiently. -[Badge version 5.5] Elasticsearch can also replace Apache Cassandra and JanusGraph for storing audit logs. +[Badge version 5.5] Elasticsearch can also replace Apache Cassandra (JanusGraph) for storing audit logs. !!! note "Elasticsearch support" Starting from version 5.3, TheHive supports Elasticsearch 8.0 and 7.x. Earlier versions only support Elasticsearch 7.x. If you want to use Elasticsearch to store your audit logs, ensure that you are using Elasticsearch 7.17 or later. @@ -521,13 +521,13 @@ By default, TheHive stores audit logs in Apache Cassandra via JanusGraph. Howeve !!! warning "Prerequisites" Regularly [back up your Elasticsearch indices](https://www.elastic.co/docs/deploy-manage/tools/snapshot-and-restore) to ensure you can recover audit logs in the event of an incident. This is critical for maintaining the integrity and availability of your data. -1. Activate audit log storage +a. Activate audit log storage - {!includes/activate-audit-log-storage-elasticsearch.md!} +{!includes/activate-audit-log-storage-elasticsearch.md!} -2. Configure index template and Index Lifecycle Management (ILM) +b. Configure index template and Index Lifecycle Management (ILM) - {!includes/configure-index-ilm-elasticsearch.md!} +{!includes/configure-index-ilm-elasticsearch.md!} ### Start the service diff --git a/docs/thehive/operations/configure-audit-logs-storage-elasticsearch.md b/docs/thehive/operations/configure-audit-logs-storage-elasticsearch.md index 248094e77d..bab1787360 100644 --- a/docs/thehive/operations/configure-audit-logs-storage-elasticsearch.md +++ b/docs/thehive/operations/configure-audit-logs-storage-elasticsearch.md @@ -4,6 +4,8 @@ You can choose to store audit logs in either Apache Cassandra via JanusGraph or in [Elasticsearch](https://www.elastic.co/enterprise-search) in TheHive. +By default, TheHive stores audit logs in Apache Cassandra via JanusGraph. + This topic provides step-by-step instructions for configuring TheHive's audit log storage in Elasticsearch for existing instances, including the option to migrate historical logs from JanusGraph. !!! info "Reasons to consider Elasticsearch" @@ -44,15 +46,15 @@ Run the `migrateAudits` script: * If you installed TheHive on a server: -``` bash -./migrateAudits --audit-from-date -c -``` + ``` bash + ./migrateAudits --audit-from-date -c + ``` * If you deployed TheHive with a Docker image: -``` bash -docker run --network host --rm -ti -v :/etc/thehive/application.conf:ro -v :/etc/thehive/logback.xml:ro strangebee/thehive:5.5.0-1-SNAPSHOT migrateAudits -Dlogback.configurationFile=/etc/thehive/logback.xml -- --audit-from-date -c /etc/thehive/application.conf -``` + ``` bash + docker run --network host --rm -ti -v :/etc/thehive/application.conf:ro -v :/etc/thehive/logback.xml:ro strangebee/thehive:5.5.0-1-SNAPSHOT migrateAudits -Dlogback.configurationFile=/etc/thehive/logback.xml -- --audit-from-date -c /etc/thehive/application.conf + ``` Available options are: diff --git a/mkdocs.yml b/mkdocs.yml index 0d684703ee..f008d69a09 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -244,6 +244,7 @@ nav: - 'Cassandra Cluster Operations': thehive/operations/cassandra-cluster.md - 'Cassandra Security Operations': thehive/operations/cassandra-security.md - 'MinIO Cluster Operations': thehive/operations/minio-cluster.md + - 'Configure Audit Log Storage in Elasticsearch': thehive/operations/configure-audit-logs-storage-elasticsearch.md - 'Backup & Restore Operations': - 'Overview': thehive/operations/backup-restore/overview.md - 'Backup Process': From c9aaedc1e675efd0580cc1fb7f30117ce00ae209 Mon Sep 17 00:00:00 2001 From: AnneLaure1307 Date: Tue, 22 Apr 2025 11:50:16 +0200 Subject: [PATCH 7/9] Fixed json format --- docs/includes/activate-audit-log-storage-elasticsearch.md | 2 +- docs/includes/configure-index-ilm-elasticsearch.md | 8 ++++---- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/includes/activate-audit-log-storage-elasticsearch.md b/docs/includes/activate-audit-log-storage-elasticsearch.md index 16bf71934f..21370df31b 100644 --- a/docs/includes/activate-audit-log-storage-elasticsearch.md +++ b/docs/includes/activate-audit-log-storage-elasticsearch.md @@ -1,6 +1,6 @@ To activate audit log storage in Elasticsearch, update your TheHive configuration file with the following settings: -```bash +```json audit { storage = elasticsearch } diff --git a/docs/includes/configure-index-ilm-elasticsearch.md b/docs/includes/configure-index-ilm-elasticsearch.md index 9eb1c95a2d..a81a94daee 100644 --- a/docs/includes/configure-index-ilm-elasticsearch.md +++ b/docs/includes/configure-index-ilm-elasticsearch.md @@ -15,7 +15,7 @@ The index template ensures consistent storage and indexing of audit logs. Default index template: -```bash +```json { "index": { "lifecycle": { @@ -33,7 +33,7 @@ ILM manages the storage, rollover, and deletion of audit logs over time, optimiz Default ILM: -```bash +```json { "policy": { "phases": { @@ -55,7 +55,7 @@ Default ILM: - Index template: - ```bash + ```json audit { storage = elasticsearch elasticsearch { @@ -70,7 +70,7 @@ Default ILM: - ILM: - ```bash + ```json { "policy": { "phases": { From e23218f481f69a89791ba2fc5add46add7e0ff8e Mon Sep 17 00:00:00 2001 From: AnneLaure1307 Date: Tue, 22 Apr 2025 12:20:10 +0200 Subject: [PATCH 8/9] Added badge --- docs/thehive/installation/step-by-step-installation-guide.md | 4 ++-- .../operations/configure-audit-logs-storage-elasticsearch.md | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/thehive/installation/step-by-step-installation-guide.md b/docs/thehive/installation/step-by-step-installation-guide.md index 872b24780f..fd47f0ad11 100644 --- a/docs/thehive/installation/step-by-step-installation-guide.md +++ b/docs/thehive/installation/step-by-step-installation-guide.md @@ -362,7 +362,7 @@ For additional configuration options, refer to: [Elasticsearch](https://www.elastic.co/elasticsearch) is a robust data indexing and search engine. TheHive uses it to manage data indices efficiently. -[Badge version 5.5] Elasticsearch can also replace Apache Cassandra (JanusGraph) for storing audit logs. + Elasticsearch can also replace Apache Cassandra (JanusGraph) for storing audit logs. !!! note "Elasticsearch support" Starting from version 5.3, TheHive supports Elasticsearch 8.0 and 7.x. Earlier versions only support Elasticsearch 7.x. If you want to use Elasticsearch to store your audit logs, ensure that you are using Elasticsearch 7.17 or later. @@ -514,7 +514,7 @@ You can configure Elasticsearch by modifying settings within the `/etc/elasticse - Similar to data and files, include indexes in the backup policy to ensure their preservation. - Remove and re-create indexes as needed. -**12. [Badge version 5.5] Optional: Configure audit log storage:** +**12. Optional: Configure audit log storage:** By default, TheHive stores audit logs in Apache Cassandra via JanusGraph. However, if your organization generates a large volume of audit logs, you can switch to Elasticsearch. Elasticsearch offers better performance, reduced latency, and advanced search capabilities, making it ideal for managing and retrieving large amounts of audit data. diff --git a/docs/thehive/operations/configure-audit-logs-storage-elasticsearch.md b/docs/thehive/operations/configure-audit-logs-storage-elasticsearch.md index bab1787360..cf3769fb61 100644 --- a/docs/thehive/operations/configure-audit-logs-storage-elasticsearch.md +++ b/docs/thehive/operations/configure-audit-logs-storage-elasticsearch.md @@ -1,6 +1,6 @@ # How to Configure Audit Log Storage in Elasticsearch -[Badge version 5.5] + You can choose to store audit logs in either Apache Cassandra via JanusGraph or in [Elasticsearch](https://www.elastic.co/enterprise-search) in TheHive. From be1bbec187f2e3fa9f361965b46eef8969f21557 Mon Sep 17 00:00:00 2001 From: AnneLaure1307 Date: Tue, 22 Apr 2025 14:01:10 +0200 Subject: [PATCH 9/9] Mentioned audit logs visibility --- .../step-by-step-installation-guide.md | 18 ++++++++++++------ ...nfigure-audit-logs-storage-elasticsearch.md | 3 +++ .../analyst-corner/cases/about-cases.md | 4 +++- 3 files changed, 18 insertions(+), 7 deletions(-) diff --git a/docs/thehive/installation/step-by-step-installation-guide.md b/docs/thehive/installation/step-by-step-installation-guide.md index fd47f0ad11..ea543ffeed 100644 --- a/docs/thehive/installation/step-by-step-installation-guide.md +++ b/docs/thehive/installation/step-by-step-installation-guide.md @@ -467,11 +467,11 @@ You can configure Elasticsearch by modifying settings within the `/etc/elasticse Set the `path.logs` and `path.data` parameters to the desired directories, such as `/var/log/elasticsearch` and `/var/lib/elasticsearch`, respectively. -**7. Optional: Configure X-Pack security:** +**7. (Optional) Configure X-Pack security:** If you're not using X-Pack security, ensure that `xpack.security.enabled` is set to `false`. -**8. Optional: Configure script allowed types:** +**8. (Optional) Configure script allowed types:** If needed, set the `script.allowed_types` parameter to specify allowed script types. @@ -514,18 +514,24 @@ You can configure Elasticsearch by modifying settings within the `/etc/elasticse - Similar to data and files, include indexes in the backup policy to ensure their preservation. - Remove and re-create indexes as needed. -**12. Optional: Configure audit log storage:** +**12. (Optional) Configure audit log storage:** -By default, TheHive stores audit logs in Apache Cassandra via JanusGraph. However, if your organization generates a large volume of audit logs, you can switch to Elasticsearch. Elasticsearch offers better performance, reduced latency, and advanced search capabilities, making it ideal for managing and retrieving large amounts of audit data. +By default, TheHive stores audit logs in Apache Cassandra via JanusGraph. However, if your organization generates a large volume of audit logs, you can switch to Elasticsearch. + +Elasticsearch offers better performance, reduced latency, and advanced search capabilities, making it ideal for managing and retrieving large amounts of audit data. + +!!! warning "Audit logs visibility" + - With Elasticsearch, audit logs retain the visibility they had at the time of creation, regardless of the current visibility of the case. This means that even if [you set restricted visibility for a case](../user-guides/analyst-corner/cases/restrict-visibility-case.md), audit logs remain visible to all users. + - With the default configuration using Apache Cassandra (JanusGraph), audit logs immediately become private if the visibility of the associated case is restricted. This means that even if audit logs were originally public, they will be hidden from all users [once the case visibility is restricted](../user-guides/analyst-corner/cases/restrict-visibility-case.md). !!! warning "Prerequisites" Regularly [back up your Elasticsearch indices](https://www.elastic.co/docs/deploy-manage/tools/snapshot-and-restore) to ensure you can recover audit logs in the event of an incident. This is critical for maintaining the integrity and availability of your data. -a. Activate audit log storage +a. Activate audit log storage. {!includes/activate-audit-log-storage-elasticsearch.md!} -b. Configure index template and Index Lifecycle Management (ILM) +b. Configure index template and Index Lifecycle Management (ILM). {!includes/configure-index-ilm-elasticsearch.md!} diff --git a/docs/thehive/operations/configure-audit-logs-storage-elasticsearch.md b/docs/thehive/operations/configure-audit-logs-storage-elasticsearch.md index cf3769fb61..6388c3a890 100644 --- a/docs/thehive/operations/configure-audit-logs-storage-elasticsearch.md +++ b/docs/thehive/operations/configure-audit-logs-storage-elasticsearch.md @@ -11,6 +11,9 @@ This topic provides step-by-step instructions for configuring TheHive's audit lo !!! info "Reasons to consider Elasticsearch" Elasticsearch is better suited for managing large volumes of audit logs. It enhances performance by efficiently handling data, reducing latency, and offering advanced search capabilities. If your organization generates a significant amount of audit logs, migrating to Elasticsearch can improve both data management and retrieval. +!!! warning "Audit logs visibility" + With Elasticsearch, audit logs retain the visibility they had at the time of creation, regardless of the current visibility of the case. This means that even if [you set restricted visibility for a case](../user-guides/analyst-corner/cases/restrict-visibility-case.md), audit logs remain visible to all users. + !!! warning "No rollback possible" Since TheHive can only use one audit storage system at a time, when you switch to Elasticsearch, the audit logs stored in JanusGraph become unavailable. TheHive doesn't support transferring audit logs back from Elasticsearch to JanusGraph. diff --git a/docs/thehive/user-guides/analyst-corner/cases/about-cases.md b/docs/thehive/user-guides/analyst-corner/cases/about-cases.md index b9b2794421..62db0c4dab 100644 --- a/docs/thehive/user-guides/analyst-corner/cases/about-cases.md +++ b/docs/thehive/user-guides/analyst-corner/cases/about-cases.md @@ -124,7 +124,9 @@ Unauthorized users can't be assigned to the case. A case with restricted visibility still triggers [configured notifications](../../organization/configure-organization/manage-notifications/about-notifications.md), regardless of who can view the case. !!! warning "Audit logs behavior" - When visibility is restricted on a case, all associated [audit logs](../../organization/about-audit-logs.md), including those created earlier, immediately become private. + When visibility is restricted on a case: + - If you're using the default configuration with Apache Cassandra (JanusGraph) to store your audit logs, all associated [audit logs](../../organization/about-audit-logs.md), including those created earlier, immediately become private. + - If you're using [Elasticsearch to store your audit logs](../../../operations/configure-audit-logs-storage-elasticsearch.md), all associated [audit logs](../../organization/about-audit-logs.md) remain public. Audit logs retain the visibility they had at the time of creation, regardless of the current visibility of the case. #### Indicators