Automated test for LLM Finetune Deepspeed demo

[abhijeet-dhumal]

Description

RHOAIENG-9185

• Automated test for LLMs fine-tuning with Ray example for Llama-2-7b and Llama-3.1-8b models

Related PRs / Depends on:
project-codeflare/codeflare-common#59
project-codeflare/codeflare-common#61
#208

How Has This Been Tested?

export CODEFLARE_TEST_OUTPUT_DIR=/home/abdhumal/abhidev/RedHatDev/distributed-workloads \
NOTEBOOK_USER_NAME=regular-user \
NOTEBOOK_USER_TOKEN=<auth-token> \
ODH_NAMESPACE=redhat-ods-applications \
NOTEBOOK_IMAGE=quay.io/modh/odh-generic-data-science-notebook@sha256:9d7f80080a453bcf7dee01b986df9ee811ee74f6f433c601a8b67d283c160547 \
AWS_ACCESS_KEY_ID=<s3-access-key> \
AWS_SECRET_ACCESS_KEY=<s3-secret-key> \
AWS_STORAGE_BUCKET=<storage-bucket-name> \
AWS_DEFAULT_REGION=<aws-default-region>

go test -timeout 15m -v ./tests/odh -run TestRayFinetuneLlmDeepspeedDemoLlama_2_7b
go test -timeout 15m -v ./tests/odh -run TestRayFinetuneLlmDeepspeedDemoLlama_31_8b

Merge criteria:

• The commits are squashed in a cohesive manner and have meaningful messages.
• Testing instructions have been added in the PR body (for PRs involving changes that are not immediately obvious).
• The developer has manually tested the changes and verified that the changes work

[Bobbins228]

What was the reasoning behind copying all of the necessary training files to this tests/odh/ray_finetune_demo? The files should exist already under examples/ray-finetune-llm-deepspeed.

This would cause any updates for the example to also be updated here

[abhijeet-dhumal]

What was the reasoning behind copying all of the necessary training files to this tests/odh/ray_finetune_demo? The files should exist already under examples/ray-finetune-llm-deepspeed.

This would cause any updates for the example to also be updated here

Hey Mark, I have updated some of the files a bit to make it compatible with paths specified during test execution and also some tweaks can be made in future for upcoming tests..
So keeping example files and test files separate seems feasible to me in this case.. please correct me if I'm wrong
Thank you for reviewing @Bobbins228 !!

[sutaakar]

/lgtm

[astefanutti]

Awesome work!

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: astefanutti

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [astefanutti]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment