Support multi node integ tests

[owaiskazi19]

Description

Support multi node integ tests.
Improvements done for CI in this PR:

1. example 1, example 2 integ test was running twice in precommit and check. Removed running integTest again in precommit
2. jacocotestReport tasks depends on test and integTest. Since gradle check already runs both, moved codeCov CI under gradle check.
3. Added multi node integration tests CI

Related Issues

Resolves ##1307

Check List

• New functionality includes testing.
• New functionality has been documented.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[vibrantvarun]

Gradle check is failing.

[yuye-aws]

@owaiskazi19 Can you update the change log?

[owaiskazi19]

Unrelated test is failing on main as well

REPRODUCE WITH: ./gradlew ':test' --tests "org.opensearch.neuralsearch.search.query.HybridCollectorManagerTests.testRescoreWithConcurrentSegmentSearch_whenMatchedDocsAndRescore_thenSuccessful" -Dtests.seed=31B75FEED86AF287 -Dtests.security.manager=false -Dtests.locale=fi-FI -Dtests.timezone=Europe/Brussels -Druntime.java=23

HybridCollectorManagerTests > testRescoreWithConcurrentSegmentSearch_whenMatchedDocsAndRescore_thenSuccessful FAILED
    java.lang.AssertionError: expected:<3> but was:<1>
        at __randomizedtesting.SeedInfo.seed([31B75FEED86AF287:BB4DF1B8A1D4DC19]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.failNotEquals(Assert.java:835)
        at org.junit.Assert.assertEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:633)
        at org.opensearch.neuralsearch.search.query.HybridCollectorManagerTests.testRescoreWithConcurrentSegmentSearch_whenMatchedDocsAndRescore_thenSuccessful(HybridCollectorManagerTests.java:1019)

[yuye-aws]

Looks good to me! Thanks for the PR @owaiskazi19 !

[yuye-aws]

Not have much context on the current flakey tests. Can @vibrantvarun help verify?

[junqiu-lei]

LGTM, thank you! @owaiskazi19

[owaiskazi19]

@yuye-aws fixed the codecov issues and improved the CI. This should be good to review.

[owaiskazi19]

REPRODUCE WITH: ./gradlew ':integTest' --tests 'org.opensearch.neuralsearch.processor.semantic.SemanticFieldProcessorIT.testSemanticFieldProcessor_withSparseModel' -Dtests.seed=BCD1D0E51A2AA11A -Dtests.security.manager=false -Dtests.locale=so -Dtests.timezone=America/Cayman -Druntime.java=23

SemanticFieldProcessorIT > testSemanticFieldProcessor_withSparseModel FAILED
    java.lang.AssertionError: failed to load the model, last task finished with status RUNNING
        at __randomizedtesting.SeedInfo.seed([BCD1D0E51A2AA11A:61A81B9B0F54104B]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.assertTrue(Assert.java:42)
        at org.opensearch.neuralsearch.BaseNeuralSearchIT.loadModel(BaseNeuralSearchIT.java:297)
        at org.opensearch.neuralsearch.processor.semantic.SemanticFieldProcessorIT.testSemanticFieldProcessor_withSparseModel(SemanticFieldProcessorIT.java:51)

need to load and wait for model before running the test similar to bwc tests

[heemin32]

Thanks for the change!

[martin-gaievski]

Looks good, thank you!