Onboard to s3 snapshots

[DarshitChanpura]

Description

Related to opensearch-project/opensearch-build#5360

Check List

• New functionality includes testing.
• New functionality has been documented.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[peterzhuamazon]

• fixed in Resolve #850 issues job-scheduler#851

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 80.14%. Comparing base (9b11665) to head (e145043).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff            @@
##               main    #4320   +/-   ##
=========================================
  Coverage     80.14%   80.14%           
- Complexity    10171    10172    +1     
=========================================
  Files           854      854           
  Lines         44219    44219           
  Branches       5113     5113           
=========================================
+ Hits          35439    35440    +1     
  Misses         6630     6630           
+ Partials       2150     2149    -1     

Flag	Coverage Δ
ml-commons	80.14% <ø> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[peterzhuamazon]

Hi @dhrubo-os @DarshitChanpura these two tests keep failling, any solutions?

Thanks!

[DarshitChanpura]

will be fixed here: #4244

[dhrubo-os]

will be fixed here: #4244

@DarshitChanpura Why it's passing for other workflows?

[peterzhuamazon]

Merging this as this is blocking #4244 which is the fix to this PR check failure issues.

Thanks.

[DarshitChanpura]

@dhrubo-os I'm not familiar with setup but do other stages invoke security related tests?

[DarshitChanpura]

I don't think so, based on this:

ml-commons/.github/workflows/CI-workflow.yml

Lines 80 to 87 in 7eb0d80

	su `id -un 1000` -c 'whoami && java -version &&
	export OPENAI_KEY=`aws secretsmanager get-secret-value --secret-id github_openai_key --query SecretString --output text` &&
	export COHERE_KEY=`aws secretsmanager get-secret-value --secret-id github_cohere_key --query SecretString --output text` &&
	echo "::add-mask::$OPENAI_KEY" &&
	echo "::add-mask::$COHERE_KEY" &&
	echo "build and run tests" && ./gradlew build -x spotlessJava &&
	echo "Publish to Maven Local" && ./gradlew publishToMavenLocal -x spotlessJava &&
	echo "Multi Nodes Integration Testing" && ./gradlew integTest -PnumNodes=3 -x spotlessJava'

but the docker workflow calls security tests:

ml-commons/.github/workflows/CI-workflow.yml

Lines 180 to 195 in 7eb0d80

	- name: Run MLCommons Test
	if: env.imagePresent == 'true'
	run: |
	security=`curl -XGET https://localhost:9200/_cat/plugins?v -u admin:${{ steps.genpass.outputs.password }} --insecure |grep opensearch-security|wc -l`
	export OPENAI_KEY=$(aws secretsmanager get-secret-value --secret-id github_openai_key --query SecretString --output text)
	export COHERE_KEY=$(aws secretsmanager get-secret-value --secret-id github_cohere_key --query SecretString --output text)
	echo "::add-mask::$OPENAI_KEY"
	echo "::add-mask::$COHERE_KEY"
	if [ $security -gt 0 ]
	then
	echo "Security plugin is available"
	./gradlew integTest -Dtests.rest.cluster=localhost:9200 -Dtests.cluster=localhost:9200 -Dtests.clustername="docker-cluster" -Dhttps=true -Duser=admin -Dpassword=${{ steps.genpass.outputs.password }} -x spotlessJava
	else
	echo "Security plugin is NOT available"
	./gradlew integTest -Dtests.rest.cluster=localhost:9200 -Dtests.cluster=localhost:9200 -Dtests.clustername="docker-cluster" -x spotlessJava
	fi

hence other CI passed but docker one fails