Make, run, cli - let's talk about setup

[choinek]

I suggest plan as follows for running scripts:

• ./cli – shortcut to scripts, e.g., cli/celery
• ./scripts/run-local.sh
• ./scripts/run-docker.sh
• ./scripts/run-celery.sh – a straightforward 1:1 redirect to Celery, enriched with -A text_extract_api.tasks

The result would be something like: ./cli celery logtool traces.

Of course, we’ll add more tools here that will also be useful for development.

Setups / Makefile

• make is a good option for installers but isn’t ideal for runtime execution. Sometimes it works, but it has the limitation of running in subshells, so you can’t fully rely on it. I’d leave basic environment setup tasks in the Makefile, while more advanced executions are moved to .sh scripts.
• For example, creating a venv isn’t feasible in a Makefile because every new command is a new subshell. As a result, tasks like running or fetching Python dependencies need to be extracted into scripts.

Summary

• scripts/... – scripts for managing and running the application.
• Makefile – basic environment configurator and initializer; it will also save basic configuration to .env, so it doesn’t need to prompt the user every time. Instead, it will use the user’s default settings (e.g., docker/local).
• cli – shortcut to scripts for developers (if we want to avoid clutter, we can add it to .gitignore, put it in scripts, and let developers copy it if they prefer, like me).

I’m also considering replacing this extended Makefile functionality with an interactive Python CLI configurator. In this case, the Makefile would essentially be limited to:

• install – installs, validates, and prepares the basic configuration. This should be the user’s first step, and it would then hand off to setup.
• setup – launches the interactive configurator.
• reset – asks twice before resetting the environment (e.g., .env, .venv, etc.).

💡 IDEAS 💡

2025-01-13

[IDEA1] Python Setup

I would go with python script, because It will be much easier to maintain than bash/make (we will use make for basic install and bash for some specific tsks)

• Interactive Menu:

  • Allow users to download non-MIT packages from external repositories with their acknowledgment and acceptance.

• Version Management:

  • Check the current published version and suggest updates.
  • Include a "keep in mind" changelog to highlight changes that might impact users (excluding API changes, as backward compatibility is easier to maintain).

• Monitoring:

  • Display the status of Celery workers (eg. number of active workers)
  • Show Redis queue status (e.g., number of waiting tasks, cache)

• Configuration:

  • Support for local, docker, and dokcer gpi envs
  • Configure multiple ollama urls, allowing easy switching between them (eg. for personal setups with multiple addresses)
    • ? Consider implementing round-robin logic for load distribution
  • Manage .env files for various settings such as prompts, storage strategies etc
  • Allow configuration of how many workers should start

2025-01-14

[IDEA2] Only one port should be propagated to the host - fastapi_app.

The rest of the services are just tools and not part of the core functionality.

I'll add easy access to the CLI (like cli/redis-cli which will be using under hood docker-compose exec it...) and remove the following from the configuration:

redis:
    image: redis:7.2.4-alpine
    container_name: redis
    ports:  # <- Remove this line
      - "6379:6379"  # <- And this line

We can also opt for a more advanced solution, as we’ll have installers that modify docker-compose based on templates.

I’ve already implemented this approach in php-multienv-library-tester so we have a ready solution for it. In Python, this will be even easier to implement.

---

[IDEA3] Universalizing Dev Environment Configuration

On the last day when I spent a lot of time making adjustments to the environment, I came to a conclusion.

I believe we should universalize every operational mode in the dev environment – i.e., GPU / non-GPU / local.

Consolidation Plan

I propose consolidating everything into a single Docker Compose template configuration, which would include:

• fastapi_app: {none/gpu/no-gpu}
• celery_worker: {none/gpu/no-gpu}
• redis: {none/yes}
• ollama: {none/gpu/no-gpu}

Configuration and Installation Workflow

During installation, based on the configuration chosen by the user:

• Docker Compose and the basic configurations will be generated.

Detailed Configuration Behavior

1. fastapi none

   • Mark in the configuration that it runs locally.
   • Service endpoints will use 127.0.0.1 with forwarded ports.

2. celery none

   • Same behavior as fastapi for local operation.

3. Redis none

   • Prompt the user to provide the address where Redis is hosted.
   • Replace the value in .env.
   • This overwrites the configuration regardless of whether fastapi and celery are running on the host or in Docker.

4. ollama none

   • Same approach as Redis; replace the value in .env.

GPU and Non-GPU Versions

• Key Difference: These versions will differ only in their configuration.

Benefits of This Approach

1. A single wizard would enable centralized management of all configurations.
2. All configurations, including the entrypoint, would be managed through environment variables (env).
3. Streamlined and simplified maintenance across operational modes.

[pkarw]

Hey @choinek the structure of the scripts - this is great, let's do it

Makefile - ok we can have it, I'm not a big fan of- it's so much... eighties? It's great for C apps but not cool for python (at least this is just my subjective POV)

But we can have it splitted as you described

Other way would be to have the init/install python scripts which could potentially give us more flexibility in the future

[pkarw]

Introducing makefile has one more downside which is that is another language contributors should know and understand. Python is the main language and I would probably say we should stick to it

[choinek]

Yeah, I understand.

I'll go with the Python way, including CLI definitions in pyproject.toml. ;-)

However, I'll keep the Makefile for old-school users. I'm just used to it, but it won't be the default and will only provide a few functionalities, like checking requirements, starting the setup, or running the application based on the currently configured mode.

[choinek]

Update: added IDEA2

[choinek]

Update: added IDEA3