Add a rollover policy to the setup plugin #269
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Policy DocumentThe setup plugin has been modified to index a document to the {
"policy": {
"policy_id": "wazuh_rollover_policy",
"description": "Example rollover policy.",
"last_updated_time": 1738947466825,
"schema_version": 21,
"error_notification": null,
"default_state": "rollover",
"states": [
{
"name": "rollover",
"actions": [
{
"retry": {
"count": 3,
"backoff": "exponential",
"delay": "1m"
},
"rollover": {
"min_doc_count": 1,
"copy_alias": false
}
}
],
"transitions": []
}
],
"ism_template": [
{
"index_patterns": [
"test-index-*"
],
"priority": 100,
"last_updated_time": 1738947466825
}
],
"user": {
"name": "admin",
"backend_roles": [
"admin"
],
"roles": [
"own_index",
"all_access"
],
"custom_attribute_names": [],
"user_requested_tenant": null
}
}
} |
Index Management index templateDuring testing we found out that our plugin was quicker to load than the Index Management one, so a template must be set up for the We took the mappings from it from here: |
Trigger the jobAn index needs to be written, which matches the index pattern the policy expects, while also being set up as the write index for the rollover alias: curl -XPUT http://localhost:9200/test-index-0000 -H 'Content-Type: application/json' -d '{"aliases":{"test-alias":{"is_write_index":true}}}' |
Check that indices are being rotatedIn order to speed up execution of the rollover policy, the following command can be issued: curl -XPUT http://localhost:9200/_cluster/settings?pretty=true -H'Content-Type: application/json' -d '{"persistent": {"plugins.index_state_management.job_interval":1}}'Now we can index new commands pointing towards our index alias: curl -XPOST http://localhost:9200/test-alias/_doc -H 'Content-Type: application/json' -d '{"field":"value"}' |
Check the policy is reckonedWe can now check whether the ISM plugin is recognizing our policy and will apply it to the right index $ curl 'http://localhost:9200/_plugins/_ism/explain?pretty'
{
"test-index-0000" : {
"index.plugins.index_state_management.policy_id" : "wazuh_rollover_policy",
"index.opendistro.index_state_management.policy_id" : "wazuh_rollover_policy",
"index" : "test-index-0000",
"index_uuid" : "v6dYvuxlS9mJ3DE_78-IyA",
"policy_id" : "wazuh_rollover_policy",
"enabled" : true
},
"total_managed_indices" : 1
}
|
4 tasks
|
When changing the OpenSearch version from 2.18.0 to 2.19.1 the solution stops working. At first the template was not configured in .opendistro-ism-config and generated problems with the mapping. Configuring the mapping directly in the .opendistro-ism-config index creation solves that problem. Although when the mapping started working the wazuh-command index and the wazuh-alerts index stopped being managed by the previously created policy. I think probably the problem is associated with the cluster reload process and this logs: |
f-galland
force-pushed
the
591-rollover-policy
branch
from
feb8ef2 to
06f9a1d
Compare
|
After the latest changes, the policy is properly loaded into the
|
|
Index rotation works: fede@tyner:~
$ curl http://localhost:9200/_cat/indices?s=index
yellow open .opendistro-job-scheduler-lock B8LqoLW1RlqQWjX06X1XUg 1 1 3 21 34.3kb 34.3kb
green open test-index-0000 ejlCKdBhS5yW9cHYl0nSRg 1 0 14 0 7.5kb 7.5kb
green open test-index-000001 CQOn5rezQxytyhsQUa_myQ 1 0 16 0 7.5kb 7.5kb
green open test-index-000002 un3eTVgaRSWfTm7I_eagWA 1 0 14 0 4kb 4kb
green open test-index-000003 e4TqINtKTDKBduiNdoNaNg 1 0 0 0 208b 208b
green open wazuh-agents 1DLMbNV4Rmuuh5QQgd4_iw 1 0 0 0 208b 208b
green open wazuh-alerts-5.x-0001 yczOfTs8R5qvJA51AlW_2A 1 0 0 0 208b 208b
green open wazuh-commands SBYfmNdhTAGrbnamLx0HnA 1 0 0 0 208b 208b
green open wazuh-custom-users wDWHbqfqS6W_7YWX4S-YEw 1 0 0 0 208b 208b
green open wazuh-cve WGjYE2YOREe5o8-iqqS9Dg 1 0 0 0 208b 208b
green open wazuh-internal-users Cd31StxfSpi21wmFtUsUPQ 1 0 0 0 208b 208b
green open wazuh-states-fim GyZn2GhuTN2tq4G-_T6hsQ 1 0 0 0 208b 208b
green open wazuh-states-inventory-hardware uIq8geHjSHCiQi_IVoo1vw 1 0 0 0 208b 208b
green open wazuh-states-inventory-hotfixes LKNc70gpSnCo37IKGw99LA 1 0 0 0 208b 208b
green open wazuh-states-inventory-networks n9v1Vl79QBC5nIDJ4BMaGg 1 0 0 0 208b 208b
green open wazuh-states-inventory-packages ETEkYdXMQxGGUKsLJoaOZg 1 0 0 0 208b 208b
green open wazuh-states-inventory-ports 4J-RDoAoQhK9R4ihxygzKw 1 0 0 0 208b 208b
green open wazuh-states-inventory-processes AqrPreE8Ruix5kGGIgYOvw 1 0 0 0 208b 208b
green open wazuh-states-inventory-system 4s6GT6U6Sl2W1dqOtpB0xw 1 0 0 0 208b 208b
green open wazuh-states-sca jTFKx-_7SSiE_CovTAcOkQ 1 0 0 0 208b 208b
green open wazuh-states-vulnerabilities AxOD1r_9QR6Ygy5ia5MbPw 1 0 0 0 208b 208b
fede@tyner:~
|
|
I've been trying to get integration tests working in order to properly test this "hack" in order to catch future breaks. Unfortunately, the log.info(Arrays.toString(client().admin().indices().prepareGetIndex().addIndices("*").get().indices()))I get an empty list. I tried overriding @Override
protected Collection<Class<? extends Plugin>> nodePlugins() {
return Collections.singletonList(SetupPlugin.class);
}But when I had this loaded, the tests would fail with a |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR is meant as a proof of concept that a rollover policy can be set up from a plugin, outside the Index Management plugin itself.
It does so by writing a policy
jsonto the.opendistro-ism-configmuch like the Index Management plugin itself would do.Issues Resolved
#435