Arkime/Cluster Prefix (feature request?)

[divinehawk]

How much work would it be to support custom Arkime prefixes? This would allow some segregation of data for retention purposes but also more efficient searches. For example site1_arkime_sessions3 instead of arkime_sessions3.

I set prefix in arkime's config and arkime behaves as expected, but this requires a separate multiviewer. Advantage is a handy dropdown in the multiviewer.
prefix=site1

What else would be required? Possibly MALCOLM_NETWORK_INDEX_PATTERN=*arkime_sessions3-*

[mmguero]

So there is some customization that can be done, but (for Malcolm, at least for now), it only applies to the Zeek and Suricata logs, not the Arkime sessions themselves. What can be done today is configured in ./config/opensearch.env

• MALCOLM_NETWORK_INDEX_PATTERN can be used to set the index pattern used by Malcolm's logstash pipeline to index network traffic logs. This includes Zeek and Suricata logs, but Arkime doesn't go through our Logstash pipeline so it does not apply to that. So you could do something like this for Zeek and Suricata logs to be put into their own indexes. Even though they'll be a different index than the regular arkime_sessions3-* ones, which is what Arkime will continue to use, the Arkime viewer should work correctly because we use this value to set queryExtraIndices in Arkime's config.ini.

# Index pattern for network traffic logs written via Logstash (e.g., Zeek logs, Suricata alerts)
MALCOLM_NETWORK_INDEX_PATTERN=foobar-*

• MALCOLM_NETWORK_INDEX_SUFFIX can be used to further customize the indexes (again, only the ones for Zeek and Suricata logs, not Arkime sessions). There are a few things you can do here, but mostly I'd suggest you include a date formatting pattern as indicated in the comment, and that it matches what we're setting for rotateIndex which is set for Malcolm in arkime.env as ARKIME_ROTATE_INDEX. So, for example, if ARKIME_ROTATE_INDEX=daily, include %{%y%m%d} at the end of MALCOLM_NETWORK_INDEX_SUFFIX. However, the other thing you can do to get further segmentation is use some other field name which will have its value expanded into the final index name as well, as the comment describes:

# Suffix used to create index to which network traffic logs are written
#   * supports Ruby strftime strings in %{}; e.g.,
#     - hourly: %{%y%m%dh%H}, twice daily: %{%P%y%m%d}, daily: %{%y%m%d}, weekly: %{%yw%U}, monthly: %{%ym%m})
#   * supports expanding dot-delimited field names in {{ }}; e.g.,
#     - {{event.provider}}%{%y%m%d}
MALCOLM_NETWORK_INDEX_SUFFIX=%{%y%m%d}

So, for example, if you did this:

MALCOLM_NETWORK_INDEX_PATTERN=foobar-*
MALCOLM_NETWORK_INDEX_SUFFIX={{event.provider}}%{%y%m%d}

Then you'd end up with indexes named like, foobar-zeek20250528, foobar-suricata20250528, etc. You wouldn't have to use event.provider, you could use some other field, but I'd try to make sure it would be one that is always going to be populated if it were me.

All of that should then be queried seamlessly in Dashboards and Arkime, however as I've mentioned, it does not change the indexes arkime itself uses when it ingests live or uploaded traffic with its capture utility. As that goes, I'm usually a bit hesitant to try to introduce new functionality upstream to Malcolm (it has happened, before, of course: the queryExtraIndices feature came from us) just for Malcolm's sake. If there is already some ability to do what you're talking about in vanilla Arkime, I'm happy to do what we need to do to surface it so it can be used in Malcolm, but TBH I'm probably not going to delve too deep into making changes to how arkime itself does its indexing.