|
| 1 | +# Deploy OPEA ChatQnA workload on OCP |
| 2 | + |
| 3 | +## Overview |
| 4 | +The workload is based on the [OPEA ChatQnA Application](https://github.com/opea-project/GenAIExamples/tree/v0.8/ChatQnA) running on Intel® Gaudi Accelerator with OpenShift and OpenShift AI. Refer to the [OPEA Generative AI Examples](https://github.com/opea-project/GenAIExamples/tree/v0.8) for more details about the OPEA workloads. |
| 5 | + |
| 6 | +**Note**: It is still under heavy development, and the updates are expected. |
| 7 | + |
| 8 | +## Prerequisites |
| 9 | +* Provisioned RHOCP cluster. Follow steps [here](/README.md#provisioning-rhocp-cluster) |
| 10 | +* The Persistent storage using NFS is ready. Refer to [documentation](https://docs.openshift.com/container-platform/4.16/storage/persistent_storage/persistent-storage-nfs.html) for the details to set it up. |
| 11 | + |
| 12 | + **Note**: Refer to [documentation](https://docs.openshift.com/container-platform/4.16/storage/index.html) for setting up other types of persistent storages. |
| 13 | +* Provisioned Intel Gaudi accelerator on RHOCP cluster. Follow steps [here](/gaudi/README.md) |
| 14 | +* RHOAI is installed. Follow steps [here](../inference/README.md/#install-rhoai) |
| 15 | +* The Intel Gaudi AI accelerator is enabled with RHOAI. Follow steps [here]((../inference/README.md/#enable-intel-gaudi-ai-accelerator-with-rhoai)) |
| 16 | +* Minio based S3 service ready for RHOAI. Follow steps [here](https://ai-on-openshift.io/tools-and-applications/minio/minio/#create-a-matching-data-connection-for-minio) |
| 17 | + |
| 18 | +## Deploy Model Serving for OPEA ChatQnA Microservices with RHOAI |
| 19 | + |
| 20 | +### Create OpenShift AI Data Science Project |
| 21 | + |
| 22 | +* Click ```Search -> Routes -> rhods-dashboard``` from the OCP web console and launch the RHOAI dashboard. |
| 23 | + |
| 24 | +* Follow the dashboard and click ```Data Science Projects``` to create a project. For example, ```OPEA-chatqna-modserving```. |
| 25 | + |
| 26 | +### Preload the models |
| 27 | + |
| 28 | +* Refer to [link](https://huggingface.co/docs/hub/en/models-downloading) and download the model [Llama2-70b-chat-hf](https://huggingface.co/meta-llama/Llama-2-70b-chat-hf). |
| 29 | + |
| 30 | +* Refer to [link](https://ai-on-openshift.io/tools-and-applications/minio/minio/#create-a-matching-data-connection-for-minio) and upload the model to minio/s3 storage. |
| 31 | + |
| 32 | +* Click ```OPEA-chatqna-modserving```, and choose ```Data Connection``` section. In the fields, add your access and secret keys from minio. Follow [link](https://ai-on-openshift.io/tools-and-applications/minio/minio/#create-a-matching-data-connection-for-minio). |
| 33 | + |
| 34 | +### Launch the Model Serving with Intel Gaudi AI Accelerator |
| 35 | + |
| 36 | +* Click on the Settings and choose ```ServingRuntime```. Copy or import the [tgi_gaudi_servingruntime.yaml](tgi-gaudi-servingruntime.yaml). The [tgi-gaudi](https://github.com/huggingface/tgi-gaudi) serving runtime is used. Follow the image below. |
| 37 | + |
| 38 | +* In the project ```OPEA-chatqna-modserving``` --> ```Models``` section and follow the image below. |
| 39 | + |
| 40 | +* The model server is now in the creation state. Once ready, the status will be updated to green and the inference endpoint can be seen. Refer to the image below. |
| 41 | + |
| 42 | +## Deploy ChatQnA Megaservice and Database |
| 43 | + |
| 44 | +### Create namespace |
| 45 | + |
| 46 | +``` |
| 47 | + oc create namespace opea-chatqna |
| 48 | +``` |
| 49 | + |
| 50 | +### Create persistent volumes |
| 51 | +The NFS is used to create the Persistent Volumes for ChatQnA MegaService to claim and use. |
| 52 | + |
| 53 | +Make sure to update NFS server IP and path in ```persistent_volumes.yaml``` before applying command below. |
| 54 | +For example: |
| 55 | +``` |
| 56 | + nfs: |
| 57 | + server: 10.20.1.2 # nfs server |
| 58 | + path: /my_nfs # nfs path |
| 59 | +``` |
| 60 | + |
| 61 | +``` |
| 62 | +$ oc apply -f https://raw.githubusercontent.com/intel/intel-technology-enabling-for-openshift/main/workloads/opea/chatqna/persistent_volumes.yaml |
| 63 | +
|
| 64 | +``` |
| 65 | + |
| 66 | +* Check that the persistent volumes are created: |
| 67 | + |
| 68 | +``` |
| 69 | +$ oc get pv |
| 70 | +NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS |
| 71 | +chatqna-megaservice-pv-0 100Mi RWO Retain Available |
| 72 | +chatqna-megaservice-pv-1 100Mi RWO Retain Available |
| 73 | +chatqna-megaservice-pv-2 100Mi RWO Retain Available |
| 74 | +
|
| 75 | +``` |
| 76 | +### Building OPEA ChatQnA MegaService Container Image |
| 77 | +``` |
| 78 | +create_megaservice_container.sh |
| 79 | +``` |
| 80 | + |
| 81 | +### Deploy Redis Vector Database Service |
| 82 | +``` |
| 83 | +$ oc apply -f https://raw.githubusercontent.com/intel/intel-technology-enabling-for-openshift/main/workloads/opea/chatqna/redis_deployment_service.yaml |
| 84 | +
|
| 85 | +``` |
| 86 | + |
| 87 | +Check that the pod and service are running: |
| 88 | + |
| 89 | +``` |
| 90 | +$ oc get pods |
| 91 | +NAME READY STATUS RESTARTS AGE |
| 92 | +redis-vector-db-6b5747bf7-sl8fr 1/1 Running 0 21s |
| 93 | +``` |
| 94 | + |
| 95 | +``` |
| 96 | +$ oc get svc |
| 97 | +NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE |
| 98 | +redis-vector-db ClusterIP 1.2.3.4 <none> 6379/TCP,8001/TCP 43s |
| 99 | +``` |
| 100 | + |
| 101 | +### Deploy ChatQnA MegaService |
| 102 | + |
| 103 | +Update the inference endpoint from the <image name> in the chatqna_megaservice_deployment. |
| 104 | + |
| 105 | +``` |
| 106 | +$ oc apply -f https://raw.githubusercontent.com/intel/intel-technology-enabling-for-openshift/main/workloads/opea/chatqna/chatqna_megaservice_deployment.yaml |
| 107 | +``` |
| 108 | + |
| 109 | +Check that the pod and service are running: |
| 110 | + |
| 111 | +``` |
| 112 | +$ oc get pods |
| 113 | +NAME READY STATUS RESTARTS AGE |
| 114 | +chatqna-megaservice-54487649b5-sgsh2 1/1 Running 0 95s |
| 115 | +``` |
| 116 | + |
| 117 | +``` |
| 118 | +$ oc get svc |
| 119 | +NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE |
| 120 | +chatqna-megaservice ClusterIP 1.2.3.4 <none> 8000/TCP 99s |
| 121 | +``` |
| 122 | + |
| 123 | +### Verify the Megaservice |
| 124 | +Use the command below: |
| 125 | + |
| 126 | +``` |
| 127 | + curl <megaservice_pod_ip>/v1/rag/chat_stream \ |
| 128 | + -X POST \ |
| 129 | + -d '{"query":"What is a constellation?"}' \ |
| 130 | + -H 'Content-Type: application/json' |
| 131 | +
|
| 132 | +``` |
0 commit comments