A/B Testing Notebook

[dmarx]

To do:

Blocking

• fix save_favorite_only behavior
• Make notebook more robust to errors, resuming
  • read outcomes.log if it exists, use it to populate running_score, SAMPLE_IDX, etc.

Potential Future Features

• richer logging
  • log time taken for generation
    • maybe writing images to gdrive is slower than writing them locally?
  • log time taken by user to choose
  • log incidence of safety mitigation trigger
  • log incidence of errors and conditions that emitted them
  • record "pass" - i.e. when user regenerates without selecting a favorite
• optionally use different seed for samples
• optionally request multiple samples per test case
• use plotly (or some such) to permit user to interactively facet histogram on combinatorial options
  • e.g. to see if there are specific prompts for which one or other test case is preferred
  • check old checkins project for a potential solution here
    • https://medium.com/plotly/introducing-jupyterdash-811f1f57c02e
    • https://github.com/dmarx/checkin/blob/master/checkin/datavis%20sandbox.ipynb
• Additional rating dimensions
  • aesthetic preference
  • relevance to prompt
  • least weird / most devoid of artifacts
  • best fit to style
  • best fit to content
  • best foreground
  • best background
  • best composition
  • best small details
  • best texture
  • most photorealistic
  • most artistic

[sonarqubecloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
No Duplication information