Elasticsearch

We are currently working on integrating the Elastic stack!

Description

From: https://www.elastic.co/products/elasticsearch

Elasticsearch is a distributed, RESTful search and analytics engine capable of solving a growing number of use cases. As the heart of the Elastic Stack, it centrally stores your data so you can discover the expected and uncover the unexpected.

Configuration

Files

• Configuration files for Elasticsearch can be found in /etc/elasticsearch/.

• Other configuration options for Elasticsearch can be found in /etc/nsm/securityonion.conf.

• By default, if total available memory is 8GB or greater, ELASTICSEARCH_HEAP in /etc/nsm/securityonion.conf is configured (during setup) to equal 25% of available memory, but no greater than 31GB.

  See https://www.elastic.co/guide/en/elasticsearch/guide/current/heap-sizing.html#compressed_oops for more details.

  You may need to adjust the value for ELASTICSEARCH_HEAP depending on your system's performance (running sudo so-elastic-restart after).

Field limit

Security Onion currently utilizes the default field limit for Elasticsearch indices (1000). If you receive error messages from Logstash, or you would simply like to increase this, you can do so with the following:

Temporary

curl -XPUT -H'Content-Type: application/json' localhost:9200/logstash-syslog-*/_settings -d'{ "index.mapping.total_fields.limit": 2000 }'

The above command would increase the field limit for the logstash-syslog-* indice(s) to 2000. Keep in mind, this setting only applies to the current index, so when the index rolls over and a new one is created, your new settings will not apply.

Persistent

If you need this change to be persistent, you can modify the "settings" stanza for the matched indices in /etc/logstash/logstash-template.json.

"settings" : {
    "number_of_replicas": 0,
    "number_of_shards": 1,
    "index.refresh_interval" : "5s",
    "index.mapping.total_fields.limit": 2000
},

Then restart Logstash with:
sudo docker stop so-logstash && sudo so-elastic-start

Note, that the change to the field limit will not occur immediately -- only upon index creation. Therefore, it is recommended to run the previously mentioned temporary command and modify the template file.

Logs

• Elasticsearch logs can be found in /var/log/elasticsearch/.
• Logging configuration can be found in /etc/elasticsearch/log4j2.properties.

Distributed

Security Onion implements distributed deployments using Elasticsearch's cross cluster search. When you run Setup and choose Sensor, it will create a local Elasticsearch instance and then configure the master server to query that instance (similar to ELSA distributed deployments). This is done by constructing an autossh tunnel from the sensor to the master server, configuring reverse port forwarding to allow the master server to connect to the local Elasticsearch instance, and updating _cluster/settings on the master server so that it will query the local Elasticsearch instance.

Removing a sensor

If you need to remove a sensor from your cross cluster search configuration, send the following to Elasticsearch on your master server (replacing "sensor2" with the actual sensor you'd like to remove and noting that null must be in square brackets):

PUT _cluster/settings
{
"persistent": {
"search": {
"remote": {
"sensor2": {
"seeds": [null]
}
}
}
}
}

You can simply copy/paste the above code (modifying as necessary) into the Console, under "Dev Tools" in Kibana, and click the green triangle. Alternatively, you could submit it to Elasticsearch via a cURL command.

Storage

All of the data Elasticsearch collects is stored under /nsm/elasticsearch/.