Spike - Viability analysis of creating ISM policies within a plugin

[AlexRuiz7]

Description

Related issues:

• Index State Management default policy wazuh#18999
• Explore how to use the ISM features from another plugin #12

One of the main requirements of the Indexer initialization plugin project is to include aliases and rollover policies to stream indices by default, as Index Management related features.

For Wazuh 5, we have identified 2 stream indices:

• wazuh-alerts data stream.
• wazuh-archives data stream.

The setup plugin (see #9) generates indices for both data streams at startup, wazuh-alerts-5.x-0001 and wazuh-archives-5.x-0001 respectively.

On this issue, we are going to create aliases and rollover policies for both data streams, defining the rollover criteria.

We have not yet found a simple way of interacting with the OpenSearch's Indexer Management plugin, which is responsible for these things. As part of this issue, we will investigate how to implement these features within our setup plugin.

Functional requirements

• The wazuh-alerts data stream is associated to an alias.
• The wazuh-alerts data stream is managed by an active rollover policy.
• The wazuh-archives data stream is associated to an alias.
• The wazuh-archives data stream is managed by an active rollover policy.
• Aliases and rollover policies are generated automatically.

Implementation restrictions

• The initialization of the index aliases and the rollover policies are the responsibility of the setup plugin.

Plan

• Spike. Investigate how the IM plugin persists such data.
• Spike. Reproduce the IM creation of policies.
• Define aliases names.
• Define rollover policies.

[mcasas993]

Investigation of how the ISM plugin persists such data

• The ismPlugin function extends the OpenSearch client by adding a namespace for managing index policies, allowing for easy interactions with the API to get, create, update, and delete policies, as well as handle associated operations like rollups and transforms.
  https://github.com/opensearch-project/index-management-dashboards-plugin/blob/main/server/clusters/ism/ismPlugin.ts

• In the ismPlugin they use API constant from "../../utils/constants" in which exists definition of parameter of Index to ISM configuration
  

export enum INDEX {
  OPENDISTRO_ISM_CONFIG = ".opendistro-ism-config",
}

• This class document and control the mapping of policy schema in ISM Plugin.

• Effectivily, after create a policy, I can search it in the .opendistro-ism-config index:

GET /.opendistro-ism-config/_search

{
  "took": 2,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 1,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": ".opendistro-ism-config",
        "_id": "first_test",
        "_score": 1,
        "_source": {
          "policy": {
            "policy_id": "first_test",
            "description": "A first test of creation of an policy",
            "last_updated_time": 1736967823609,
            "schema_version": 21,
            "error_notification": null,
            "default_state": "pre_alias_removed",
            "states": [
              {
                "name": "pre_alias_removed",
                "actions": [
                  {
                    "retry": {
                      "count": 3,
                      "backoff": "exponential",
                      "delay": "1h"
                    },
                    "alias": {
                      "actions": [
                        {
                          "remove": {
                            "aliases": [
                              "commands"
                            ]
                          }
                        }
                      ]
                    }
                  }
                ],
                "transitions": []
              }
            ],
            "ism_template": [],
            "user": {
              "name": "admin",
              "backend_roles": [
                "admin"
              ],
              "roles": [
                "own_index",
                "all_access"
              ],
              "custom_attribute_names": [],
              "user_requested_tenant": null
            }
          }
        }
      }
    ]
  }
}

UPDATE

• The alias to do the rollover is applied in the settings of index that we want to manage with the policy

PUT /our_index/_settings
{
  "index": {
    "plugins": {
      "index_state_management": {
        "rollover_alias": "name_of_alias_to_rollover"
      }
    }
  }
}

[mcasas993]

Complete test of policy to rollover

1. Test in dashboard

### 1. Test in dashboard

Policy created to rollover alias

GET /.opendistro-ism-config/_search/
{
  "query": {
    "match": {
      "_id":"policy_rollover_6"
    }
  }
}

{
  "took": 1,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 1,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": ".opendistro-ism-config",
        "_id": "policy_rollover_6",
        "_score": 1,
        "_source": {
          "policy": {
            "policy_id": "policy_rollover_6",
            "description": "Created first the policy",
            "last_updated_time": 1737399485357,
            "schema_version": 21,
            "error_notification": null,
            "default_state": "created",
            "states": [
              {
                "name": "created",
                "actions": [
                  {
                    "retry": {
                      "count": 3,
                      "backoff": "exponential",
                      "delay": "1m"
                    },
                    "rollover": {
                      "min_doc_count": 3,
                      "copy_alias": true
                    }
                  }
                ],
                "transitions": [
                  {
                    "state_name": "rollovered",
                    "conditions": {
                      "min_doc_count": 3
                    }
                  }
                ]
              },
              {
                "name": "rollovered",
                "actions": [],
                "transitions": []
              }
            ],
            "ism_template": [
              {
                "index_patterns": [
                  "another-*"
                ],
                "priority": 1,
                "last_updated_time": 1737382578280
              }
            ],
            "user": {
              "name": "admin",
              "backend_roles": [
                "admin"
              ],
              "roles": [
                "own_index",
                "all_access"
              ],
              "custom_attribute_names": [],
              "user_requested_tenant": null
            }
          }
        }
      }
    ]
  }
}

Index created and with an alias associated

Policy applied with the alias specified

GET /another-rollover/_settings

{
  "another-rollover-index-00001": {
    "settings": {
      "index": {
        "replication": {
          "type": "DOCUMENT"
        },
        "refresh_interval": "1s",
        "number_of_shards": "1",
        "plugins": {
          "index_state_management": {
            "rollover_alias": "another-rollover",
            "auto_manage": "false"
          }
        },
        "provided_name": "another-rollover-index-00001",
        "creation_date": "1737399216111",
        "number_of_replicas": "0",
        "uuid": "WSeQeDH7S7a9LQTs0qqTSw",
        "version": {
          "created": "136347827"
        }
      }
    }
  }
}

Create documents in the test index

Create five documents like this:

POST /another-rollover/_doc
{
  "name": "Another Example",
  "price": 19.99,
  "description": "We are such stuff as dreams are made on"
}

Results of the policy run

Result in another-rollover-index-00001 index

Result in another-rollover-index-00002 index

2. Test in manually in DevOp

2. Test in manually in DevOp

Policy created to rollover alias

POST .opendistro-ism-config/_doc/manual_policy_rollover_3
     {
      "policy": {
          "description": "Created third manual rollover the policy",
          "last_updated_time": 1737399485357,
          "schema_version": 21,
          "error_notification": null,
          "default_state": "Initial",
          "states": [
            {
              "name": "Initial",
              "actions": [
                {
                  "retry": {
                    "count": 3,
                    "backoff": "exponential",
                    "delay": "1m"
                  },
                  "rollover": {
                    "min_doc_count": 3,
                    "copy_alias": true
                  }
                }
              ],
              "transitions": [
                {
                  "state_name": "Rollovered",
                  "conditions": {
                    "min_doc_count": 3
                  }
                }
              ]
            },
            {
              "name": "Rollovered",
              "actions": [],
              "transitions": []
            }
          ],
          "ism_template": [
            {
              "index_patterns": [
                "manual-*"
              ],
              "priority": 1,
              "last_updated_time": 1737382578280
            }
          ],
          "user": {
            "name": "admin",
            "backend_roles": [
              "admin"
            ],
            "roles": [
              "own_index",
              "all_access"
            ],
            "custom_attribute_names": [],
            "user_requested_tenant": null
          }
        }
      }

Index created and with an alias associated

Policy applied with the alias specified

PUT /manual-0001/_settings
{
  "index": {
    "plugins": {
      "index_state_management": {
        "rollover_alias": "manual"
      }
    }
  }
}

Policy applied with the alias specified

POST .opendistro-ism-config/_doc/
{
    "managed_index": {
      "name": "manual-0001",
      "enabled": false,
      "index": "manual-0001",
      "index_uuid": "OG3zRya0RoiKpjbxyJTbbA",
      "schedule": {
        "interval": {
          "start_time": 1737494828282,
          "period": 5,
          "unit": "Minutes"
        }
      },
      "last_updated_time": 1737496659085,
      "enabled_time": null,
      "policy_id": "manual_policy_rollover_3",
      "policy_seq_no": 2510,
      "policy_primary_term": 3,
      "policy": {
            "description": "Created third manual rollover the policy",
            "last_updated_time": 1737399485357,
            "schema_version": 21,
            "error_notification": null,
            "default_state": "Initial",
            "states": [
              {
                "name": "Initial",
                "actions": [
                  {
                    "retry": {
                      "count": 3,
                      "backoff": "exponential",
                      "delay": "1m"
                    },
                    "rollover": {
                      "min_doc_count": 3,
                      "copy_alias": true
                    }
                  }
                ],
                "transitions": [
                  {
                    "state_name": "Rollovered",
                    "conditions": {
                      "min_doc_count": 3
                    }
                  }
                ]
              },
              {
                "name": "Rollovered",
                "actions": [],
                "transitions": []
              }
            ],
            "ism_template": [
              {
                "index_patterns": [
                  "manual-*"
                ],
                "priority": 1,
                "last_updated_time": 1737382578280
              }
            ],
            "user": {
              "name": "admin",
              "backend_roles": [
                "admin"
              ],
              "roles": [
                "own_index",
                "all_access"
              ],
              "custom_attribute_names": [],
              "user_requested_tenant": null
            }
          },
      "change_policy": null,
      "jitter": 0.6
    }
}

Results of the policy run

Results of the policy run in manual-0001

[mcasas993]

Complete test of policy to rollover based on the previous issue

Test the issue steps

Test the issue steps

Applied an ISM policy for rollover as follows:

1. Template modification

• Edit /etc/filebeat/wazuh-template.json and add the following line inside the settings block:

  ```json
  "index.plugins.index_state_management.rollover_alias": "test"
  ```
  

• Restart wazuh-manager

  systemctl restart wazuh-manager.service

2. ISM rollover and alias policy

   • Push ISM policy to the Wazuh indexer cluster:

     "min_size": "250mb" for testing purposes only

curl -XPOST  -k -u admin:$admin_pass "https://127.0.0.1:9200/.opendistro-ism-config/_doc/MANUAL_wazuh_rollover_policy" -H 'Content-Type: application/json' -d'
{
    "policy": {
      "policy_id": "MANUAL_wazuh_rollover_policy",
      "description": "Wazuh rollover and alias policy created directly on index .opendistro-ism-config",
      "last_updated_time": 1737572429671,
      "schema_version": 21,
      "error_notification": null,
      "default_state": "active",
      "states": [
        {
          "name": "active",
          "actions": [
            {
              "retry": {
                "count": 3,
                "backoff": "exponential",
                "delay": "1m"
              },
              "rollover": {
                "min_size": "250mb",
                "copy_alias": false
              }
            }
          ],
          "transitions": []
        }
      ],
      "ism_template": [
        {
          "index_patterns": [
            "wazuh-alerts-*"
          ],
          "priority": 50,
          "last_updated_time": 1737572429671
        }
      ],
      "user": {
        "name": "admin",
        "backend_roles": [
          "admin"
        ],
        "roles": [
          "own_index",
          "all_access"
        ],
        "custom_attribute_names": [],
        "user_requested_tenant": null
      }
    }
}'

* Create initial index and quick-start the rolling process:
```bash
curl -k -u admin:$admin_pass -X PUT "https://127.0.0.1:9200/%3Ctest-1.x-%7Bnow%2Fd%7D-000001%3E?pretty" -H 'Content-Type: application/json' -d'

{
"aliases": {
"test": {
"is_write_index": true
}
}
}'

**RESULT**
Error: ""message": "Missing rollover_alias index setting [index=test-1.x-2025.01.22-000001]""

![Image](https://github.com/user-attachments/assets/3656f2b7-477a-49a9-88b7-05473217eaac)

3. Try with a the template to the index
   * Create a template to the index
    ![Image](https://github.com/user-attachments/assets/f731e246-fe23-4b65-8b86-31639e0d027f)
   
   *Delete the manage of the policy in the index
    ![Image](https://github.com/user-attachments/assets/dcaf651d-3970-4190-8c9d-2564b24eda91)

   *Delete the index
    In Dev Tools: DELETE /test-1.x-2025.01.22-000001

    * Create initial index and quick-start the rolling process:
    ```bash
    curl -k -u admin:$admin_pass -X PUT "https://127.0.0.1:9200/%3Ctest-1.x-%7Bnow%2Fd%7D-000001%3E?pretty" -H 'Content-Type: application/json' -d'
{
  "aliases": {
    "test": {
      "is_write_index": true
    }
  }
}'

RESULT
Error: ""message": "Missing rollover_alias index setting [index=test-1.x-2025.01.22-000001]""

4. Put the settiing in the index

PUT /test/_settings
{
  "index": {
    "plugins": {
      "index_state_management": {
        "rollover_alias": "test"
      }
    }
  }
}

RESULT
Error: ""message": "{
"cause": "Rollover alias [test] can point to multiple indices, found duplicated alias [[test]] in index template [test]",
"message": "Failed to rollover index [index=test-1.x-2025.01.22-000001]"
}""

[AlexRuiz7]

Simple ISM configuration

Below there is a simple ISM configuration to automatically roll over the wazuh-commands indices.

PUT _template/wazuh-commands
{
  "index_patterns": [
    "wazuh-commands*"
  ],
  "mappings": {
    "date_detection": false,
    "dynamic": "true",
    "properties": {
      "@timestamp": {
        "type": "date"
      },
      "agent": {
        "properties": {
          "groups": {
            "ignore_above": 1024,
            "type": "keyword"
          }
        }
      },
      "command": {
        "properties": {
          "action": {
            "properties": {
              "args": {
                "type": "object"
              },
              "name": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "version": {
                "ignore_above": 1024,
                "type": "keyword"
              }
            }
          },
          "order_id": {
            "ignore_above": 1024,
            "type": "keyword"
          },
          "request_id": {
            "ignore_above": 1024,
            "type": "keyword"
          },
          "result": {
            "properties": {
              "code": {
                "type": "short"
              },
              "data": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "message": {
                "ignore_above": 1024,
                "type": "keyword"
              }
            }
          },
          "source": {
            "ignore_above": 1024,
            "type": "keyword"
          },
          "status": {
            "ignore_above": 1024,
            "type": "keyword"
          },
          "target": {
            "properties": {
              "id": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "type": {
                "ignore_above": 1024,
                "type": "keyword"
              }
            }
          },
          "timeout": {
            "type": "short"
          },
          "user": {
            "ignore_above": 1024,
            "type": "keyword"
          }
        }
      },
      "delivery_timestamp": {
        "type": "date"
      }
    }
  },
  "order": 1,
  "settings": {
    "index": {
      "number_of_replicas": "0",
      "number_of_shards": "1",
      "query.default_field": [
        "command.source",
        "command.target.type",
        "command.status",
        "command.action.name"
      ],
      "refresh_interval": "5s",
      "plugins.index_state_management.rollover_alias": "wazuh-commands"
    }
  },
  "version": 500
}

PUT _plugins/_ism/policies/wazuh_rollover_policy
{
  "policy": {
    "description": "Wazuh rollover and alias policy",
    "default_state": "active",
    "states": [
      {
        "name": "active",
        "actions": [
          {
            "rollover": {
              "min_index_age": "5m"
            }
          }
        ]
      }
    ],
    "ism_template": {
      "index_patterns": ["wazuh-commands*"],
      "priority": "50"
    }
  }
}

PUT wazuh-commands-0001
{
  "aliases": {
    "wazuh-commands": {
      "is_write_index": true
    }
  }
}

[mcasas993]

Test ISM rollover configuration by creating the policy directly on the index .opendistro-ism-config

1. Add index template for the indices with the setting "plugins.index_state_management.rollover_alias": "<alias>".

   Upload template

   PUT _template/wazuh-commands
   {
     "index_patterns": [
       "wazuh-commands*"
     ],
     "mappings": {
       "date_detection": false,
       "dynamic": "true",
       "properties": {
         "@timestamp": {
           "type": "date"
         },
         "agent": {
           "properties": {
             "groups": {
               "ignore_above": 1024,
               "type": "keyword"
             }
           }
         },
         "command": {
           "properties": {
             "action": {
               "properties": {
                 "args": {
                   "type": "object"
                 },
                 "name": {
                   "ignore_above": 1024,
                   "type": "keyword"
                 },
                 "version": {
                   "ignore_above": 1024,
                   "type": "keyword"
                 }
               }
             },
             "order_id": {
               "ignore_above": 1024,
               "type": "keyword"
             },
             "request_id": {
               "ignore_above": 1024,
               "type": "keyword"
             },
             "result": {
               "properties": {
                 "code": {
                   "type": "short"
                 },
                 "data": {
                   "ignore_above": 1024,
                   "type": "keyword"
                 },
                 "message": {
                   "ignore_above": 1024,
                   "type": "keyword"
                 }
               }
             },
             "source": {
               "ignore_above": 1024,
               "type": "keyword"
             },
             "status": {
               "ignore_above": 1024,
               "type": "keyword"
             },
             "target": {
               "properties": {
                 "id": {
                   "ignore_above": 1024,
                   "type": "keyword"
                 },
                 "type": {
                   "ignore_above": 1024,
                   "type": "keyword"
                 }
               }
             },
             "timeout": {
               "type": "short"
             },
             "user": {
               "ignore_above": 1024,
               "type": "keyword"
             }
           }
         },
         "delivery_timestamp": {
           "type": "date"
         }
       }
     },
     "order": 1,
     "settings": {
       "index": {
         "number_of_replicas": "0",
         "number_of_shards": "1",
         "query.default_field": [
           "command.source",
           "command.target.type",
           "command.status",
           "command.action.name"
         ],
         "refresh_interval": "5s",
         "plugins.index_state_management.rollover_alias": "wazuh-commands"
       }
     },
     "version": 500
   }

2. Index policy

   Index policy in `.opendistro-ism-config`

   POST .opendistro-ism-config/_doc/wazuh_rollover_policy
   {
             "policy": {
               "policy_id": "wazuh_rollover_policy",
               "description": "Wazuh rollover and alias policy",
               "last_updated_time": 1738255639727,
               "schema_version": 21,
               "error_notification": null,
               "default_state": "active",
               "states": [
                 {
                   "name": "active",
                   "actions": [
                     {
                       "retry": {
                         "count": 3,
                         "backoff": "exponential",
                         "delay": "1m"
                       },
                       "rollover": {
                         "min_index_age": "5m",
                         "copy_alias": false
                       }
                     }
                   ],
                   "transitions": []
                 }
               ],
               "ism_template": [
                 {
                   "index_patterns": [
                     "wazuh-commands*"
                   ],
                   "priority": 50,
                   "last_updated_time": 1738255639727
                 }
               ],
               "user": {
                 "name": "admin",
                 "backend_roles": [
                   "admin"
                 ],
                 "roles": [
                   "own_index",
                   "all_access"
                 ],
                 "custom_attribute_names": [],
                 "user_requested_tenant": null
               }
             }
         }
   

3. Create initial index with alias

   Create index alias

   PUT wazuh-commands-0001
   {
     "aliases": {
       "wazuh-commands": {
         "is_write_index": true
       }
     }
   }

RESULTS

Policy created

Policy managed indexes

Index created

Configuration of active index with alias set

[mcasas993]

Define aliases name

The best options to the alias are the name for the index without any aggregation of number, date or anything:

• wazuh-alerts
• wazuh-commands

Because the prefix "wazuh" indicates that these are indexes generated for Wazuh automatically.

[mcasas993]

Define rollover policies

After careful consideration and analysis of the requirements from both the previous issues and industry best practices, I think the best options to implement a rollover policy for Wazuh data streams are the following conditions:

1. Maximum Document Count: 20,000,000,000 documents
2. Maximum Shard Size: 25 GB per shard
3. Maximum Index Age: 7 days

Justification:

• Maximum Document Count: Setting a maximum document count of 20 billion ensures we adhere to the limitations of OpenSearch while accommodating hyper-scaling scenarios.

• Maximum Shard Size: A maximum shard size of 25 GB aligns with best practices for performance. Shard sizes in this range (between 10-50 GB) help balance the need for efficient metadata management while allowing for effective data distribution across the cluster. This approach mitigates issues related to excessive small shards, which could lead to exhausting JVM memory.

• Maximum Index Age: Limiting the index age to 7 days prevents the cluster from being cluttered with indices that do not, in fact, enhance performance. If the shard size limit or document count limit has not been reached, there is not much benefit in generating numerous indexes just because of their age.

Policies

Policy for wazuh-commands*

"policy": {
    "policy_id": "wazuh-command_rollover_policy",
    "description": "Wazuh-command rollover and alias policy",
    "last_updated_time": XXXXX,
    "schema_version": XXXX,
    "error_notification": null,
    "default_state": "active",
    "states": [
      {
        "name": "active",
        "actions": [
          {
            "retry": {
              "count": 3,
              "backoff": "exponential",
              "delay": "1m"
            },
            "rollover": {
              "min_doc_count": 200000000,
              "min_index_age": "7d",
              "min_primary_shard_size": "25gb",
              "copy_alias": false
            }
          }
        ],
        "transitions": []
      }
    ],
    "ism_template": [
      {
        "index_patterns": [
          "wazuh-commands*"
        ],
        "priority": 50,
        "last_updated_time": XXXXX
      }
    ]
  }

Policy for wazuh-alerts*

"policy": {
    "policy_id": "wazuh-alerts_rollover_policy",
    "description": "Wazuh-alerts rollover and alias policy",
    "last_updated_time": XXXXX,
    "schema_version": XXXX,
    "error_notification": null,
    "default_state": "active",
    "states": [
      {
        "name": "active",
        "actions": [
          {
            "retry": {
              "count": 3,
              "backoff": "exponential",
              "delay": "1m"
            },
            "rollover": {
              "min_doc_count": 200000000,
              "min_index_age": "7d",
              "min_primary_shard_size": "25gb",
              "copy_alias": false
            }
          }
        ],
        "transitions": []
      }
    ],
    "ism_template": [
      {
        "index_patterns": [
          "wazuh-alerts*"
        ],
        "priority": 50,
        "last_updated_time": XXXXX
      }
    ]
  }

[f-galland]

We were able to get the policy to be applied from our setup plugin:

• Add a rollover policy to the setup plugin #269

Comments in the PR explain how to try that out.

[f-galland]

A viable approach has been demonstrated on the PoC in the linked PR.

However, a new official OpenSearch plugin was released in the meantime which may cover our needs:

• https://github.com/opensearch-project/opensearch-system-templates/tree/2.19.2.0

I'm running a quick PoC to test whether this can be used to load an ISM policy on startup.

[f-galland]

Upon closer inspection, the opensearch-system-templates plugin, seems to be geared towards setting up so-called Index Contexts, which are a set of customized settings and mappings to improve performance of indices.

As of today, the code of the plugin still looks somewhat minimal:

• https://github.com/opensearch-project/opensearch-system-templates/tree/main/src/main/java/org/opensearch/system/applicationtemplates

I don't think this will be useful at all to load our own ISM policies, since the policies themselves are documents.

[f-galland]

I've been trying to use the opensearch-index-management spi to try to avoid us the hassle of loading the template and creating the index .opendistro-ism-config, but the jar is not separately available as in the case of the job-scheduler.
We will resort to doing these things manually from our plugin for now, though we might revisit the issue later.