feat: Setup Github Action workflow file for PromptFoo #307

fg-nava · 2025-05-09T20:29:10Z

Ticket

https://navalabs.atlassian.net/browse/DST-964

Changes

Added GitHub Action workflow to run promptfoo evaluations on PRs
Integrated with Google Sheets for test case inputs and evaluation outputs
Configured promptfoo config that uses dev environment endpoint
Added automated PR comments showing evaluation results, a shareable link to PromptFoo result, and a link to the Google Sheet
- Note: PromptFoo shareable links are only accessible to users who have been invited to register for PromptFoo and are signed into their PromptFoo account. Please ensure you are signed in to the NavaLabs team on PromptFoo to view the evaluation results through the shareable links.

Testing

Workflow runs on PRs when changes are made to:

app/src/chat_api.py
app/src/chat_engine.py
app/src/generate.py
app/promptfooconfig.ci.yaml

Local Testing with Act

Install act if not already installed:

brew install act

Create a .secrets file in the project root with the following environment variables:

GITHUB_TOKEN (for local testing only): Classic Personal Access Token (PAT) with 'repo' scope (Settings > Developer settings > Personal access tokens > Tokens (classic))
GOOGLE_CREDENTIALS_JSON: Service account JSON for Google Sheets access
- For your first time setting this value use this command: echo "GOOGLE_CREDENTIALS_JSON='$(cat /path/to/service-account.json | jq -c .)'" >> .secrets
OPENAI_API_KEY: OpenAI API key for LLM evaluation
GOOGLE_SHEET_INPUT_URL: URL to input test cases sheet
GOOGLE_SHEET_OUTPUT_URL: URL to output results sheet
PROMPTFOO_API_KEY: API key for PromptFoo cloud features

Run the workflow locally:

act pull_request -W .github/workflows/promptfoo-googlesheet-evaluation.yml \
  -s GITHUB_TOKEN -s GOOGLE_CREDENTIALS_JSON -s OPENAI_API_KEY \
  --artifact-server-path /tmp/artifacts \
  --container-architecture linux/amd64 -v

GitHub Testing

Required secrets are configured in GitHub at: Settings > Secrets and variables > Actions (exclude GITHUB_TOKEN, this was only for local testing)

Example workflow run: GitHub Actions Run

Example automatic PR comment:

Preview environment for app

♻️ Environment destroyed ♻️

…egration

github-actions · 2025-05-09T20:34:52Z

☂️ Python Coverage

current status: ✅

Overall Coverage

Lines	Covered	Coverage	Threshold	Status
4264	3890	91%	80%	🟢

New Files

No new covered files...

Modified Files

No covered modified files...

updated for commit: 4d1908b by action🐍

github-actions · 2025-05-09T22:29:58Z

Promptfoo Evaluation Results

Success	Failure	Total	Pass Rate
0	0	0	NaN%

View detailed results in Google Sheets

Run promptfoo view --id null locally to view interactive results

github-actions · 2025-05-09T22:38:36Z

Promptfoo Evaluation Results

Success	Failure	Total	Pass Rate
13	0	13	100.00%

View detailed results in Google Sheets

Run promptfoo view --id null locally to view interactive results

github-actions · 2025-05-10T00:54:31Z

Promptfoo Evaluation Results

Success	Failure	Total	Pass Rate
13	0	13	100.00%

View detailed results in Google Sheets

» View eval results «

github-actions · 2025-05-10T01:10:43Z

Promptfoo Evaluation Results

Success	Failure	Total	Pass Rate
51	11	62	82.26%

View detailed results in Google Sheets

» View eval results in CI console «

github-actions · 2025-05-10T01:24:34Z

Promptfoo Evaluation Results

Success	Failure	Total	Pass Rate
47	15	62	75.81%

View detailed results in Google Sheets

» View eval results «

github-actions · 2025-05-10T01:45:28Z

Promptfoo Evaluation Results

Success	Failure	Total	Pass Rate
51	11	62	82.26%

View detailed results in Google Sheets

» View eval results «

github-actions · 2025-05-10T01:50:24Z

Promptfoo Evaluation Results

Success	Failure	Total	Pass Rate
51	11	62	82.26%

View detailed results in Google Sheets

» View eval results «

github-actions · 2025-05-12T18:07:21Z

Promptfoo Evaluation Results

Success	Failure	Total	Pass Rate
0	0	0	NaN%

View detailed results in Google Sheets

» View eval results in CI console «

github-actions · 2025-05-12T18:13:17Z

Promptfoo Evaluation Results

Success	Failure	Total	Pass Rate
51	11	62	82.26%

View detailed results in Google Sheets

» View eval results «

KevinJBoyer

LGTM!

KevinJBoyer · 2025-05-12T18:32:22Z

.github/workflows/promptfoo-googlesheet-evaluation.yml

+
+      - name: Create unique ID generator
+        run: |
+          cat > /tmp/generateUniqueId.js << 'EOF'


I'd consider moving the /docs/app/generateUniqueId.js out of the docs directory and into somwhere like app/src/evaluation and then referencing it here

we're not using /docs/app/generateUniqueId.js here, the script used in promptfoo.ci.config.yaml is written within the Create unique ID generator step.

I can still move the JS file or just add it as a snippet to the README

Sorry that suggestion wasn't clear -- what I mean is move the file and then read from it here (or reference it directly in promptfoo.ci.config.yaml) so that you don't have to embed Javascript inside of a .yml file

Sounds good, will move the script here and reference it

.github/workflows/promptfoo-googlesheet-evaluation.yml

Co-authored-by: Kevin Boyer <kevinboyer@navapbc.com>

github-actions · 2025-05-12T18:47:41Z

Promptfoo Evaluation Results

Success	Failure	Total	Pass Rate
53	9	62	85.48%

View detailed results in Google Sheets

» View eval results «

github-actions · 2025-05-12T19:46:15Z

Promptfoo Evaluation Results

Success	Failure	Total	Pass Rate
0	0	0	NaN%

View detailed results in Google Sheets

» View eval results in CI console «

github-actions · 2025-05-12T19:53:15Z

Promptfoo Evaluation Results

Success	Failure	Total	Pass Rate
53	9	62	85.48%

View detailed results in Google Sheets

» View eval results «

yoomlam · 2025-05-12T19:56:11Z

.github/workflows/promptfoo-googlesheet-evaluation.yml

+          sed -i "s|GOOGLE_SHEET_INPUT_URL|${{ env.GOOGLE_SHEET_INPUT_URL }}|g" /tmp/promptfooconfig.processed.yaml
+          sed -i "s|GOOGLE_SHEET_OUTPUT_URL|${{ env.GOOGLE_SHEET_OUTPUT_URL }}|g" /tmp/promptfooconfig.processed.yaml
+          sed -i "s|CHATBOT_INSTANCE_URL|${{ env.CHATBOT_INSTANCE_URL }}|g" /tmp/promptfooconfig.processed.yaml


Try the envsubst command instead?

Nice suggestion. Thank you.

github-actions · 2025-05-12T20:17:00Z

Promptfoo Evaluation Results

Success	Failure	Total	Pass Rate
53	9	62	85.48%

View detailed results in Google Sheets

» View eval results «

feat: Setup Github Action workflow file for PromptFoo-GoogleSheet int…

a18d5b7

…egration

fg-nava added 4 commits May 9, 2025 14:55

fix: remove cat from json verify

bf50d7d

fix: upgrade actions/cache version to v4

3dabfa0

fix: gha linter errors

8b60452

save sheet urls to secrets, add PR comment

cf3b36a

save sheet urls to secrets, add PR comment

40ae25a

fg-nava added 5 commits May 9, 2025 15:50

add share link

fcd24a1

fix: lint errors

dbeb07f

fix: auth check

62322a4

fix: auth check;

9f47e09

fix: auth check;

88aa6fa

fix: continue on error

5400362

fix: exit code

db9cc71

fix: cleanup inline comments and outputs

10a92fb

fix: add no-cache and suppress npm install warnings

8be97ce

fg-nava marked this pull request as ready for review May 12, 2025 14:34

fg-nava requested a review from a team May 12, 2025 14:34

fg-nava changed the title ~~feat: Setup Github Action workflow file for PromptFoo-GoogleSheet int…~~ feat: Setup Github Action workflow file for PromptFoo May 12, 2025

fg-nava added 2 commits May 12, 2025 11:03

fix: add workflow_dispatch for inputs

4ab6e4d

fix: remove defaults from dispatch inputs

12fce13

fix: add fallback to chatbot instance url

454ab3e

KevinJBoyer approved these changes May 12, 2025

View reviewed changes

Update .github/workflows/promptfoo-googlesheet-evaluation.yml

95c892e

Co-authored-by: Kevin Boyer <kevinboyer@navapbc.com>

fix: move config deps to app/promptfoo

6fc8709

fix: path to js script

b7d1cbb

yoomlam reviewed May 12, 2025

View reviewed changes

try use envsubst in nunjucks

4d1908b

fg-nava merged commit 99212e5 into main May 12, 2025
13 checks passed

fg-nava deleted the feat/promptfoo-github-action branch May 12, 2025 20:30

feat: Setup Github Action workflow file for PromptFoo #307

feat: Setup Github Action workflow file for PromptFoo #307

Uh oh!

Conversation

fg-nava commented May 9, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Ticket

Changes

Testing

Local Testing with Act

GitHub Testing

Preview environment for app

Uh oh!

github-actions bot commented May 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

☂️ Python Coverage

Overall Coverage

New Files

Modified Files

Uh oh!

github-actions bot commented May 9, 2025

Promptfoo Evaluation Results

Uh oh!

github-actions bot commented May 9, 2025

Promptfoo Evaluation Results

Uh oh!

github-actions bot commented May 10, 2025

Promptfoo Evaluation Results

Uh oh!

github-actions bot commented May 10, 2025

Promptfoo Evaluation Results

Uh oh!

github-actions bot commented May 10, 2025

Promptfoo Evaluation Results

Uh oh!

github-actions bot commented May 10, 2025

Promptfoo Evaluation Results

Uh oh!

github-actions bot commented May 10, 2025

Promptfoo Evaluation Results

Uh oh!

github-actions bot commented May 12, 2025

Promptfoo Evaluation Results

Uh oh!

github-actions bot commented May 12, 2025

Promptfoo Evaluation Results

Uh oh!

KevinJBoyer left a comment

Choose a reason for hiding this comment

Uh oh!

KevinJBoyer May 12, 2025

Choose a reason for hiding this comment

Uh oh!

fg-nava May 12, 2025

Choose a reason for hiding this comment

Uh oh!

fg-nava May 12, 2025

Choose a reason for hiding this comment

Uh oh!

KevinJBoyer May 12, 2025

Choose a reason for hiding this comment

Uh oh!

fg-nava May 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions bot commented May 12, 2025

Promptfoo Evaluation Results

Uh oh!

github-actions bot commented May 12, 2025

Promptfoo Evaluation Results

Uh oh!

github-actions bot commented May 12, 2025

Promptfoo Evaluation Results

Uh oh!

yoomlam May 12, 2025

Choose a reason for hiding this comment

Uh oh!

fg-nava May 12, 2025

fg-nava commented May 9, 2025 •

edited by github-actions bot

Loading

github-actions bot commented May 9, 2025 •

edited

Loading

fg-nava May 12, 2025 •

edited

Loading