Metadata tests can not be executed when there is an unrelated directory named "tests".

[lkies]

Describe the bug
As the title says, if you have a notebook with tests in the metadata they can not be executed if a directory named "tests" exists in the same location as the notebook.

To Reproduce
Steps to reproduce the behavior:

1. Create and assignment with a test stored in notebook metadata (the default settings).
2. Place directory named "tests" in the same directory as the notebook.
3. Attempt to check an exercise with grader.check(...).
4. Testing failes with a FileNotFoundError because the test can not be found.

Expected behavior
The check should run as usual, reading the test data from the notebook metadata.

Versions
Python: 3.12.8
Otter: 6.1.3

Additional context
The reason for this behavior is that resolve_test_info() check first if there are tests stored in a "tests" directory and only resorts to metadata tests if this directory does not exists. See here. Since the source of the tests to be checked is mistakenly resolved to a non-existant file in the unrelated "tests" directory, the check fails with a FileNotFoundError.

I am not familiar with the codebase, but maybe the resolution priority could be different or more thourough so that it does not assume that the tests is in the "tests" directory just because it exists" or by reversing the priority, first checking if the concrete test exists in the notebook metadata.

The Notebook class constructor has a test_dir option and if this is set to a non-existant directory everything works as expected again. So a rather simple fix could be to expose the test_dir location as a configuration option and then add the respective argument to the initialization cell.

I am willing to work on this myself, if someone tells me which of the approaches I should take.

[chrispyles]

Thanks for the bug report. The behavior of preferring the tests directory over the notebook metadata tests is important for grading inside the grading container since the notebook needs to read the tests from the tests directory (which always exists in the grading container and contains the hidden tests). So I think a good way to do this would be to prefer metadata tests unless grading mode is enabled and then prefer the configured tests directory. Any PR would need to add a test to check that this behavior (preferring metadata tests in non-grading mode and preferring tests dir in grading mode) works correctly.

Feel free to send a PR if you have time and I'll review it, or I can take a look at later (not sure when).