Skip to content

Commit 407ded0

Browse files
AnneLaure1307AnneLaure1307
authored
Merge pull request #191 from StrangeBeeCorp/DOC-129
DOC-129
2 parents f036d99 + d4a79d8 commit 407ded0

File tree

7 files changed

+236
-6
lines changed

7 files changed

+236
-6
lines changed

‎docs/includes/activate-audit-log-storage-elasticsearch.md

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
To activate audit log storage in Elasticsearch, update your TheHive configuration file with the following settings:
2+
3+
```json
4+
audit {
5+
storage = elasticsearch
6+
}
7+
```

‎docs/includes/configure-index-ilm-elasticsearch.md

Lines changed: 118 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,118 @@
1+
Configure the default index template and Index Lifecycle Management (ILM) for `thehive-audits` using the following settings:
2+
3+
* `index-name`
4+
* `health-request-timeout`
5+
* `create.ext.number_of_replicas`
6+
* `create.ext.number_of_shards`
7+
* `request-timeout`
8+
* `hot.rollover.maxAge`
9+
* `hot.rollover.maxSize`
10+
* `delete.minAge`
11+
12+
!!! warning "Index name"
13+
The index name shouldn't be the same as the one used in JanusGraph. If you don’t configure a specific name, TheHive uses the JanusGraph index name appended with the suffix `-audits`.
14+
15+
#### Index template
16+
17+
The index template ensures consistent storage and indexing of audit logs.
18+
19+
Default index template:
20+
21+
```json
22+
{
23+
"index": {
24+
"lifecycle": {
25+
"name": "thehive-audits"
26+
},
27+
"number_of_shards": "1",
28+
"number_of_replicas": "1"
29+
}
30+
}
31+
```
32+
33+
#### ILM
34+
35+
TheHive automatically generates an Index Lifecycle Management (ILM) policy based on your configuration settings. ILM manages the storage, rollover, and deletion of audit logs over time to optimize long-term storage.
36+
37+
Default generated ILM:
38+
39+
```json
40+
{
41+
"policy": {
42+
"phases": {
43+
"hot": {
44+
"min_age": "0ms",
45+
"actions": {
46+
"set_priority": {
47+
"priority": 100
48+
}
49+
}
50+
}
51+
}
52+
}
53+
}
54+
```
55+
56+
!!! example "Example"
57+
To create a new index every day or when it reaches 1 GB, and retain it for 7 days, use the following configuration:
58+
59+
- Index template:
60+
61+
```json
62+
audit {
63+
storage = elasticsearch
64+
elasticsearch {
65+
hot.rollover {
66+
maxAge = 1d
67+
maxSize = 1g
68+
}
69+
delete.minAge = 7d
70+
}
71+
}
72+
```
73+
74+
- The generated ILM:
75+
76+
```json
77+
{
78+
"policy": {
79+
"phases": {
80+
"hot": {
81+
"min_age": "0ms",
82+
"actions": {
83+
"set_priority": {
84+
"priority": 100
85+
},
86+
"rollover": {
87+
"max_primary_shard_size": "1gb",
88+
"max_age": "1d"
89+
}
90+
}
91+
},
92+
"delete": {
93+
"min_age": "7d",
94+
"actions": {
95+
"delete": {
96+
"delete_searchable_snapshot": true
97+
}
98+
}
99+
}
100+
}
101+
}
102+
}
103+
```
104+
105+
!!! warning "API calls changes"
106+
After migrating to Elasticsearch, the following Query API calls no longer work with audit logs stored in Elasticsearch:
107+
108+
* `listAudit`
109+
* `listAuditFromObject`
110+
* `getAudit`
111+
112+
Instead, use the following API endpoints:
113+
114+
* `/api/v1/audit/list`
115+
* `/api/v1/audit/count`
116+
* `/api/v1/audit/$ID`
117+
118+
These API calls are currently available but not yet officially documented and may change in future releases.

‎docs/thehive/installation/step-by-step-installation-guide.md

Lines changed: 28 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -363,11 +363,13 @@ For additional configuration options, refer to:
363363

364364
[Elasticsearch](https://www.elastic.co/elasticsearch) is a robust data indexing and search engine. TheHive uses it to manage data indices efficiently.
365365

366+
<!-- md:version 5.5 --> Elasticsearch can also replace Apache Cassandra (JanusGraph) for storing audit logs.
367+
366368
!!! note "Elasticsearch support"
367-
Starting from version 5.3, TheHive supports Elasticsearch 8.0 and 7.x. Earlier versions only support Elasticsearch 7.x.
369+
Starting from version 5.3, TheHive supports Elasticsearch 8.0 and 7.x. Earlier versions only support Elasticsearch 7.x. If you want to use Elasticsearch to store your audit logs, ensure that you are using Elasticsearch 7.17 or later.
368370

369371
!!! note "OpenSearch support"
370-
Starting from version 5.3, TheHive supports [OpenSearch](https://opensearch.org/) for advanced use cases.
372+
Starting from version 5.3, TheHive supports [OpenSearch](https://opensearch.org/) for advanced use cases, except for audit log storage.
371373

372374
!!! warning "Elasticsearch authentication permissions"
373375
The account used for authenticating with Elasticsearch must have [the `manage` cluster privilege](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-privileges.html#privileges-list-cluster). This is a mandatory configuration for TheHive to function correctly. If you are using an existing Elasticsearch instance, ensure it complies with your internal security policies, as certain configurations might be incompatible. Additionally, the account must have [the `all` indices privilege](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-privileges.html#privileges-list-indices), specifically for the `thehive*` indices.
@@ -466,11 +468,11 @@ You can configure Elasticsearch by modifying settings within the `/etc/elasticse
466468

467469
Set the `path.logs` and `path.data` parameters to the desired directories, such as `/var/log/elasticsearch` and `/var/lib/elasticsearch`, respectively.
468470

469-
**7. Optional: Configure X-Pack security:**
471+
**7. (Optional) Configure X-Pack security:**
470472

471473
If you're not using X-Pack security, ensure that `xpack.security.enabled` is set to `false`.
472474

473-
**8. Optional: Configure script allowed types:**
475+
**8. (Optional) Configure script allowed types:**
474476

475477
If needed, set the `script.allowed_types` parameter to specify allowed script types.
476478

@@ -514,6 +516,28 @@ You can configure Elasticsearch by modifying settings within the `/etc/elasticse
514516
* Similar to data and files, include indexes in the backup policy to ensure their preservation.
515517
* Remove and re-create indexes as needed.
516518

519+
**12. <!-- md:version 5.5 --> (Optional) Configure audit log storage:**
520+
521+
By default, TheHive stores audit logs in Apache Cassandra via JanusGraph. However, if your organization generates a large volume of audit logs, you can switch to Elasticsearch.
522+
523+
Elasticsearch offers better performance, reduced latency, and advanced search capabilities, making it ideal for managing and retrieving large amounts of audit data.
524+
525+
!!! warning "Audit logs visibility"
526+
527+
* With Elasticsearch, audit logs retain the visibility they had at the time of creation, regardless of the current visibility of the case. This means that even if [you set restricted visibility for a case](../user-guides/analyst-corner/cases/case-visibility/restrict-visibility-case.md), audit logs remain visible to all users who originally had access.
528+
* With the default configuration using Apache Cassandra (JanusGraph), audit logs immediately become private if the visibility of the associated case is restricted. This means that even if audit logs were originally public, they will be hidden from all users [once the case visibility is restricted](../user-guides/analyst-corner/cases/case-visibility/restrict-visibility-case.md).
529+
530+
!!! warning "Prerequisites"
531+
Regularly [back up your Elasticsearch indices](https://www.elastic.co/docs/deploy-manage/tools/snapshot-and-restore) to ensure you can recover audit logs in the event of an incident. This is critical for maintaining the integrity and availability of your data.
532+
533+
a. Activate audit log storage.
534+
535+
{!includes/activate-audit-log-storage-elasticsearch.md!}
536+
537+
b. Optional: Configure index template and Index Lifecycle Management (ILM).
538+
539+
{!includes/configure-index-ilm-elasticsearch.md!}
540+
517541
### Start the service
518542

519543
=== "DEB"

‎docs/thehive/operations/configure-audit-logs-storage-elasticsearch.md

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
# How to Configure Audit Log Storage in Elasticsearch
2+
3+
<!-- md:version 5.5 -->
4+
5+
By default, TheHive stores audit logs in Apache Cassandra via JanusGraph. However, if your organization generates a large volume of audit logs, you can switch to [Elasticsearch](https://www.elastic.co/enterprise-search).
6+
7+
This topic provides step-by-step instructions for configuring TheHive's audit log storage in Elasticsearch for existing instances, including the option to migrate historical logs from JanusGraph.
8+
9+
For new installations, refer to the [Step-by-Step Guide](../installation/step-by-step-installation-guide.md#fontawesome-solid-list-elasticsearch) for instructions.
10+
11+
!!! info "Reasons to consider Elasticsearch"
12+
Elasticsearch is better suited for managing large volumes of audit logs. It enhances performance by efficiently handling data, reducing latency, and offering advanced search capabilities. If your organization generates a significant amount of audit logs, migrating to Elasticsearch can improve both data management and retrieval.
13+
14+
!!! danger "No rollback possible"
15+
Since TheHive can only use one audit storage system at a time, when you switch to Elasticsearch, the audit logs stored in JanusGraph become unavailable. TheHive doesn't support transferring audit logs back from Elasticsearch to JanusGraph.
16+
17+
!!! warning "Audit logs visibility"
18+
With Elasticsearch, audit logs retain the visibility they had at the time of creation, regardless of the current visibility of the case. This means that even if [you set restricted visibility for a case](../user-guides/analyst-corner/cases/case-visibility/restrict-visibility-case.md), audit logs remain visible to all users who originally had access.
19+
20+
## Prerequisites
21+
22+
### Use Elasticsearch 7.17 or later
23+
24+
Make sure you're using Elasticsearch version 7.17 or later to guarantee compatibility with TheHive's audit log storage. OpenSearch isn't supported for audit logs.
25+
26+
### Back up Elasticsearch indices
27+
28+
Regularly [back up your Elasticsearch indices](https://www.elastic.co/docs/deploy-manage/tools/snapshot-and-restore) to ensure you can recover audit logs in the event of an incident. This is critical for maintaining the integrity and availability of your data.
29+
30+
## Step 1: Activate audit log storage in Elasticsearch
31+
32+
{!includes/activate-audit-log-storage-elasticsearch.md!}
33+
34+
## (Optional) Step 2: Configure index template and Index Lifecycle Management (ILM)
35+
36+
{!includes/configure-index-ilm-elasticsearch.md!}
37+
38+
## (Optional) Step 3: Migrate historical audit logs to Elasticsearch
39+
40+
TheHive provides a script to assist with migrating historical audit logs from JanusGraph to Elasticsearch. Run the script only if you need to transfer historical audit logs to Elasticsearch.
41+
42+
!!! warning "JanusGraph audit log deletion"
43+
By default, when migrating audit logs from JanusGraph to Elasticsearch, the logs are deleted from JanusGraph. If you wish to keep the audit logs in JanusGraph while migrating to Elasticsearch, you can prevent their deletion by using a specific option.
44+
45+
Run the `migrateAudits` script:
46+
47+
* If you installed TheHive on a server:
48+
49+
``` bash
50+
./migrateAudits --audit-from-date <date> -c </etc/thehive/application.conf>
51+
```
52+
53+
* If you deployed TheHive with a Docker image:
54+
55+
``` bash
56+
docker run --network host --rm -ti -v </path/to/your/application.conf>:/etc/thehive/application.conf:ro -v </path/to/your/logback.xml>:/etc/thehive/logback.xml:ro strangebee/thehive:5.5.0-1-SNAPSHOT migrateAudits -Dlogback.configurationFile=/etc/thehive/logback.xml -- --audit-from-date <date> -c /etc/thehive/application.conf
57+
```
58+
59+
Available options are:
60+
61+
```
62+
-v, --version: Displays the version of the migration tool.
63+
-h, --help: Displays help information.
64+
-c, --config <file> : Specifies the global configuration file for TheHive.
65+
-y, --transaction-pagesize <value>: Defines the page size for each transaction during the migration.
66+
-n, --no-deletion: Prevents the deletion of audit logs from JanusGraph after they are migrated to Elasticsearch.
67+
--audit-from-date <date>: Migrates only the audit logs created from the specified date. The date format is yyyyMMdd[HH[mm[ss]]] or yyyy/MM/dd HH:mm:ss.
68+
--audit-until-date <date>: Migrates only the audits created until the specified date. The date format is yyyyMMdd[HH[mm[ss]]] or yyyy/MM/dd HH:mm:ss.
69+
```
70+
71+
<h2>Next steps</h2>
72+
73+
* [Back up and restore](../operations/backup-restore/overview.md)
74+
* [Monitoring](monitoring.md)
75+
* [Troubleshooting](troubleshooting.md)

‎docs/thehive/user-guides/analyst-corner/cases/about-cases.md

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -126,7 +126,9 @@ Unauthorized users can't be assigned to the case.
126126
A case with restricted visibility still triggers [configured notifications](../../organization/configure-organization/manage-notifications/about-notifications.md), regardless of who can view the case.
127127

128128
!!! warning "Audit logs behavior"
129-
When visibility is restricted on a case, all associated [audit logs](../../organization/about-audit-logs.md), including those created earlier, immediately become private.
129+
When visibility is restricted on a case:
130+
- If you're using the default configuration with Apache Cassandra (JanusGraph) to store your audit logs, all associated [audit logs](../../organization/about-audit-logs.md), including those created earlier, immediately become private.
131+
- If you're using [Elasticsearch to store your audit logs](../../../operations/configure-audit-logs-storage-elasticsearch.md), all associated [audit logs](../../organization/about-audit-logs.md) remain public. Audit logs retain the visibility they had at the time of creation, regardless of the current visibility of the case.
130132

131133
#### Indicators
132134

‎docs/thehive/user-guides/analyst-corner/cases/create-a-new-case.md

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -112,7 +112,9 @@ Several options are offered to create a case in TheHive:
112112

113113
4. Select **Confirm**.
114114

115-
## Create a case from a [MISP event](../../../administration/misp-integration/about-misp-integration.md)
115+
## Create a case from a MISP event
116+
117+
See [About MISP Integration](../../../administration/misp-integration/about-misp-integration.md) to learn about how MISP integrates with TheHive.
116118

117119
!!! info "Data transfer"
118120
When creating a case from a MISP event, data from the event, such as observables, is automatically transferred to the case.

‎mkdocs.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -50,6 +50,7 @@ plugins:
5050
- exclude-search:
5151
exclude:
5252
- includes/*
53+
- thehive/operations/configure-audit-logs-storage-elasticsearch.md
5354
- awesome-pages
5455
- glightbox
5556
- mkdocs-video:
@@ -404,6 +405,7 @@ nav:
404405
- 'Cassandra Cluster Operations': thehive/operations/cassandra-cluster.md
405406
- 'Cassandra Security Operations': thehive/operations/cassandra-security.md
406407
- 'MinIO Cluster Operations': thehive/operations/minio-cluster.md
408+
# - 'Configure Audit Log Storage in Elasticsearch': thehive/operations/configure-audit-logs-storage-elasticsearch.md
407409
- 'Backup & Restore Operations':
408410
- 'Overview': thehive/operations/backup-restore/overview.md
409411
- 'Backup Process':

0 commit comments

Comments
 (0)