Skip to content

BUG: Ollama test suite in CI is a false positive. It succeeds even though the tests don't run to completion #58

Open
@iambenkay

Description

@iambenkay

The model names provided in the CI for Ollama differ from the names provided in the test suite yet the ollama integration test passes.

This behaviour is incorrect. Ideally, the test should fail if this kind of inconsistency exists and it should never pass.

ci.yaml

Image

test-llm/lib.rs

Image

Result in Ollama integration test suite (while passing successfully)

Image

You can rework the tests to only pass when no errors are returned from the model so that the CI can fail and signal an issue.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions