[INFRA] Run multi node integration test on Github CI

[junqiu-lei]

What's the issue?

Currently, the neural-search plugin's integration tests only run on a single node environment. However, in production, OpenSearch operates as a distributed cluster with multiple nodes. This gap in testing means we can't verify critical distributed functionality like cross-node search operations.

What's the solution?

Add multi-node integration testing to our GitHub Actions workflow. This will help catch distributed system issues early and ensure the plugin works reliably in production environments.

Example: https://github.com/opensearch-project/index-management/blob/main/.github/workflows/multi-node-test-workflow.yml

[martin-gaievski]

it's a good suggestion, testing everything on 1 node is definitely not close to prod. What is you plan in term of test coverage for these approach? I think we can use a subset of existing tests, otherwise CI will take forever because it's gonna be all we do have today + same test that will take longer due to multiple nodes.

[heemin32]

Shouldn't it be covered in bwc test? Otherwise, we can run every tests in two nodes cluster instead of single node cluster. I cannot think of the case where test is failing only in single node cluster while it succeed in multi node cluster.

[junqiu-lei]

@martin-gaievski I think we can just run the all tests in two nodes cluster.

@heemin32, while BWC tests use multiple nodes, their primary focus is compatibility across versions, not necessarily all distributed aspects of the current version. Issues like race conditions or specific internode communication problems might only surface in dedicated multi-node tests for the current codebase.

[owaiskazi19]

@junqiu-lei raised #1320, if you had something similar in mind

[junqiu-lei]

@junqiu-lei raised #1320, if you had something similar in mind

Thanks @owaiskazi19, yes #1320 can cover the case I mentioned here

[heemin32]

This multi-node issue primarily stems from the serialization and deserialization of requests between nodes. It currently affects the stats APIs and neural query functionality. In the case of neural queries, simply increasing the number of nodes isn't sufficient. To ensure proper request distribution, the number of index shards should either match the number of nodes or be set to at least 2, so that requests can target multiple nodes instead of just one.