Help build haystack

[rgstephens]

I'd like to build haystack locally and contribute. Following the instructions in CONTRIBUTING.md but I must be missing something.

Here's what I did working off main:

cd haystack/
conda create --name haystack-source python=3.9
conda activate haystack-source
pip install --upgrade pip
pip install -e '.[dev]'
pre-commit install
pytest . --document_store_type="memory"

pytest fails with the following:

ImportError while loading conftest '/Users/greg/Dev/haystack/haystack/test/conftest.py'.
test/conftest.py:75: in <module>
    _optional_component_not_installed("test", "test", ie)
haystack/utils/import_utils.py:56: in _optional_component_not_installed
    raise ImportError(
E   ImportError: Failed to import 'test', which is an optional component in Haystack.
E   Run 'pip install 'farm-haystack[test]'' to install the required dependencies and make this component available.
E   (Original error: No module named 'weaviate')

But the docs say: The InMemoryDocument store is a very good starting point as it doesn't require any external resource

Then did as suggested in the error message:

pip install 'farm-haystack[test]'
# also tried
pip install '.[test]'

This did not fix the issue. Installed:

pip install weaviate-client==3.9.0

This gets me further but now have many new dependency errors. I must be missing an install step. I reviewed the pyproject.toml. It references hatchling so I tried installing hatch and running hatch build but I still have dependency issues.

[rgstephens]

I did pip install -e '.[all]' and got much further but still have issues.

ImportError while importing test module '/Users/greg/Dev/haystack/haystack/rest_api/test/test_rest_api.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../../../miniforge3/envs/haystack-source/lib/python3.9/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
E   ModuleNotFoundError: No module named 'test.test_rest_api'
_________________________________________________________________________________________ ERROR collecting ui/test/test_ui_utils.py __________________________________________________________________________________________
ImportError while importing test module '/Users/greg/Dev/haystack/haystack/ui/test/test_ui_utils.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../../../miniforge3/envs/haystack-source/lib/python3.9/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
E   ModuleNotFoundError: No module named 'test.test_ui_utils'
====================================================================================================== warnings summary ======================================================================================================

[rgstephens]

I found a recent post that lead me to the the command to install the rest_api and ui dependencies as well as how to start a local instance with gunicorn.

pip install -e rest_api/
pip install -e ui/

[julian-risch]

Hello @rgstephens sorry to hear that setting up a development environment with Haystack is troublesome. We are aware of that and identified simplifying the installation of Haystack as a focus topic for our Q1 2023 roadmap. The steps you describe look good to me, so the error is on our side.

Starting with the v1.12 release, there is no [test] installation extra anymore because we removed it with the PR
#3679. The dependency you need for development and running the tests are unfortunately not included when you install Haystack with [dev], for example through pip install -e '.[dev]'. Until we fixed that, you can run

[docstores,audio,crawler,preprocessing,ocr,ray,dev] instead of just [dev] when you run pip install -e '.[dev]' to solve the problem. It will install all the dependencies you need to run core Haystack tests. They are in the test/ directory so when you want to run just those you missed the step to got into this directory.
If you want to run also ui and rest api tests you need to add pip install -e rest_api/ and pip install -e ui/ as you described and go to the respective directories.

So when you are on the current main branch, the full list of steps to run the core Haystack tests is the following. I will update the CONTRIBUTING.md file accordingly.

cd haystack/
conda create --name haystack-source python=3.9
conda activate haystack-source
pip install --upgrade pip
pip install -e '.[docstores,audio,crawler,preprocessing,ocr,ray,dev]'
pre-commit install
cd test/
pytest . --document_store_type="memory"

if you want to run the rest_api tests you can do the following to install the required dependencies (including httpx dependency only needed to run the tests)

cd rest_api/
pip install -e '.[dev]'
pytest . 

and run the ui tests in the same way.
We seem to missing httpx dependency from the rest_api dependencies, so you would need to also run pip install httpx before you can run the rest_api tests. I will also fix this.