Merge pull request #312 from Climate-REF/fix-integration

Author: lewisjared

.github/workflows/ci.yaml

@@ -70,12 +70,12 @@ jobs:
           python-version: ${{ matrix.python-version }}
       - name: Check importable without extras
         run: |
-          uv run --package climate_ref scripts/test-install.py climate_ref
-          uv run --package climate_ref_core scripts/test-install.py climate_ref_core
-          uv run --package climate_ref_celery scripts/test-install.py climate_ref_celery
-          uv run --package climate_ref_ilamb scripts/test-install.py climate_ref_ilamb
-          uv run --package climate_ref_esmvaltool scripts/test-install.py climate_ref_esmvaltool
-          uv run --package climate_ref_pmp python scripts/test-install.py climate_ref_pmp
+          uv run --isolated --with-editable packages/climate-ref --with typer --no-project scripts/test-install.py climate_ref
+          uv run --isolated --with-editable packages/climate-ref-core --with typer --no-project scripts/test-install.py climate_ref_core
+          uv run --isolated --with-editable packages/climate-ref-celery --with typer --no-project scripts/test-install.py climate_ref_celery
+          uv run --isolated --with-editable packages/climate-ref-ilamb --with typer --no-project scripts/test-install.py climate_ref_ilamb
+          uv run --isolated --with-editable packages/climate-ref-esmvaltool --with typer --no-project scripts/test-install.py climate_ref_esmvaltool
+          uv run --isolated --with-editable packages/climate-ref-pmp --with typer --no-project scripts/test-install.py climate_ref_pmp
 
   check-build:
     runs-on: ubuntu-latest

.github/workflows/install-pypi.yaml

@@ -61,6 +61,9 @@ jobs:
       uses: actions/checkout@v4
       with:
         ref: ${{ env.INSTALLED_VERSION }}
+        # Windows can't clone the entire directory due to filename length
+        sparse-checkout: |
+          scripts
     - name: Test installation
       run: |
         which python
@@ -112,6 +115,8 @@ jobs:
       uses: actions/checkout@v4
       with:
         ref: ${{ env.INSTALLED_VERSION }}
+        sparse-checkout: |
+          scripts
     - name: Test installation
       run: |
         which python

Makefile

@@ -172,6 +172,7 @@ fetch-test-data:  ## Download any data needed by the test suite
 
 .PHONY: fetch-ref-data
 fetch-ref-data:  ## Download reference data needed by providers and (temporarily) not in obs4mips
+	uv run ref datasets fetch-data --registry esmvaltool
 	uv run ref datasets fetch-data --registry ilamb
 	uv run ref datasets fetch-data --registry iomb
 

changelog/312.docs.md

@@ -0,0 +1 @@
+Updated documentation to include more information about concepts within the REF.

docs/development.md

@@ -34,11 +34,6 @@ make virtual-environment
 mkdir $PWD/.ref
 uv run ref config list > $PWD/.ref/ref.toml
 export REF_CONFIGURATION=$PWD/.ref
-
-# Download some test data and ingest the sample datasets.
-make fetch-test-data
-uv run ref datasets ingest --source-type cmip6 $PWD/tests/test-data/sample-data/CMIP6/
-uv run ref datasets ingest --source-type obs4mips $PWD/tests/test-data/sample-data/obs4MIPs/
 ```
 
 `uv` will create a virtual Python environment in the directory `.venv` containing
@@ -55,17 +50,46 @@ If there are any issues, the messages from the `Makefile` should guide you
 through. If not, please raise an issue in the
 [issue tracker](https://github.com/Climate-REF/climate-ref/issues).
 
-### Running your first `solve`
+### Ingesting datasets
 
-If you want to run the sample data through the whole pipeline, you need to download
-reference data, but note that the reference data is several Gigabytes in size.
+The REF requires datasets, both reference and model, to be ingested into the database.
+These ingested datasets are then used to solve for what executions are available and require running.
 
-```shell
-# Download reference data which is not (yet) included in obs4mips
-make fetch-ref-data
+We have a consistent set of decimated sample data that is used for testing.
+These can be ingested using the following command:
+
+```bash
+make fetch-test-data
+uv run ref datasets ingest --source-type cmip6 $PWD/tests/test-data/sample-data/CMIP6/
+uv run ref datasets ingest --source-type obs4mips $PWD/tests/test-data/sample-data/obs4REF/
+```
+
+Additional reference datasets can be fetched by following the instructions [here](introduction/required_datasets.md).
+The Obs4REF step is not required as we have already ingested these datasets above.
+
+### Creating provider environments
+
+The REF uses a number of different providers to run the diagnostics.
+Some of these providers may require an additional conda environment to be created before running.
+
+```bash
+uv run ref providers create-env
+```
+
+The created environments and their locations can be viewed using the command:
+
+```bash
+uv run ref providers list
 ```
 
-After that, you can let the REF calculate all included metrics for the sample data.
+### Running your first `solve`
+
+Once you have ingested some sample data and created any required environments,
+you can run your first `solve` command.
+
+A `solve` will take the ingested datasets and the providers declared in the configuration, and determine
+which new executions are required.
+
 Note that this will take a while to run.
 
 ```shell
@@ -74,7 +98,8 @@ uv run ref solve
 
 Afterwards, you can check the output of `uv run ref executions list-groups` to see if metrics
 were evaluated successfully, and if they were, you find the results in the
-`.ref/results` folder. Don't worry too much if some executions are failing for you,
+`$PWD/.ref/results` folder.
+Don't worry too much if some executions are failing for you,
 things are still in active development at the moment.
 
 ### Pip editable installation
@@ -104,14 +129,17 @@ Some metric providers can use their own conda environments.
 The REF can manage these for you,
 using a bundled version of [micromamba](https://github.com/mamba-org/micromamba-releases).
 
-A new environment for a metric provider can be created with the following command:
+The conda environments for the registered providers can be created with the following command:
 
 ```bash
-ref --log-level=info providers create-env --provider esmvaltool
+ref --log-level=info providers create-env
 ```
 
 A new environment will be created automatically for each conda-based metric provider when it is first used,
 if one does not already exist.
+This can cause issues if the environment is created on a node that doesn't have internet access,
+or if a race condition occurs when multiple processes try to create the environment at the same time.
+
 
 /// admonition | Note
 

docs/hackathon.md

@@ -45,66 +45,11 @@ The metrics that are currently available in the REF are relatively limited so no
 It is recommended to read through the [developer documentation](development.md) to get an understanding of the REF and how we collaborate.
 
 For those interested in learning more about the REF,
-we recommend reading the [Architecture design document](background/architecture.md).
+we recommend reading the [Introduction section](introduction/index.md) [Architecture design document](background/architecture.md).
 This outlines the design of the REF and provides some background about the project.
 
-## Brief technical overview of the REF
 
-The REF is a Python package that provides a framework for running benchmarking climate models. It is designed to be flexible and extensible, allowing users to define their own metrics and ingest their own climate datasets.
-
-The REF consists of 4 main steps, which are shown in the diagram below:
-
-```mermaid
-flowchart LR
-    Ingest --> Solve
-    Solve --> Execute
-    Execute --> Visualise
-```
-
-* **Ingest** Ingesting datasets that can be used for analysis into a local database. This includes support for CMIP6, Obs4MIPs, and other observational datasets.
-* **Solve** Using the ingested datasets and the data requirements from the metrics, the REF determines which metrics need to be executed.
-* **Execute** The metrics are executed and the results collated. We support multiple different ways of running metrics that may be useful for different use cases.
-* **Visualise** The results of the metrics are visualised. This can be in the form of plots, tables, or other outputs.
-
-### Diagnostics
-At the core of the REF is the [Diagnostic][climate_ref_core.diagnostics.Diagnostic] protocol.
-This protocol defines the common interface that all metrics must implement.
-A diagnostic defines the different datasets that a metric requires (see [dataset-selection](how-to-guides/dataset-selection.py)), and how to calculate a number of metric value from them.
-These metric values can be either scalar or timeseries data.
-How a metric is actually calculated depends on which metrics provider the metric comes from.
-
-The rest of the complexity that comes from figuring out which datasets to use, how to run the diagnostic, and how to visualise the results is handled by the REF.
-
-### Diagnostic Providers
-Diagnostics are grouped into packages, one for each of the diagnostic providers selected for the CMIP7 AFT.
-These diagnostic packages (ESMValTool, ILAMB and PMP) each have different ways of calculating diagnostics.
-
-For some diagnostic providers (ILAMB and ESMValTool),
-an additional conda environment will be required
-to run the metrics locally.
-This is still a work in progress ([#117](https://github.com/Climate-REF/climate-ref/pull/117))
-and is expected to be available to use by the time of the hackathon.
-
-### Ouput
-
-The output of a diagnostic execution can be a range of different outcomes:
-
-* Scalar values
-* Timeseries
-* Plots
-* Tables
-* HTML reports
-
-To support the capture of these different outputs in a standardised way,
-the REF uses the [Earth System Metrics and Diagnostics Standards (EMDS)](https://github.com/Earth-System-Diagnostics-Standards/EMDS).
-A REF-specific extension to this format will be developed as
-part of this process.
-
-These outputs will be ingested into a database
-and made available through an API/web interface or CLI tool.
-This is still early work so input into how we can expose the results in a meaningful way is welcome.
-
-### Compute Engine
+## Deployment environments
 
 The REF is designed to be run on a local machine,
 but we are also working on support for running the REF on cloud-based systems and HPC systems.

docs/installation.md

@@ -11,7 +11,8 @@ Each of these provider-specific environments are decoupled to allow for potentia
 This uses a bundled version of the [micromamba](https://github.com/mamba-org/micromamba-releases)
 to create and manage the environment so no additional dependencies are required.
 
-## PyPI
+## Installing `climate-ref`
+### PyPI
 
 You can install `climate-ref` using `pip`:
 
@@ -42,7 +43,7 @@ The actual execution occurs within a provider-specific conda environment.
 
 
 
-## Conda
+### Conda
 
 /// admonition | conda-forge
     type: warning
@@ -62,7 +63,7 @@ A modern alternative to using `conda` as package a manager is [Pixi](https://pix
 Pixi uses the same packages as `conda` but is faster and can create reproducible environments out of the box.
 
 
-## Docker
+### Docker
 
 For production use, we recommend using the `climate-ref` Docker image to provide a consistent environment for running the REF.
 Not all users may support docker directly, instead requiring the use of [Apptainer](https://apptainer.org/docs/user/latest/) to provide some additional isolation.
@@ -93,7 +94,7 @@ bash scripts/initialise-docker.sh
 docker-compose up
 ```
 
-## From Source
+### From Source
 
 To install `climate-ref` from the source code, clone the repository and install it:
 
@@ -104,3 +105,50 @@ make virtual-environment
 ```
 
 See the [development documentation](development.md) for more information on how to contribute to the project.
+
+
+## Installing metric provider dependencies
+
+/// admonition | Windows support
+   type: warning
+
+Window's doesn't support some of the packages required by the metrics providers,
+so we only support MacOS and Linux.
+Windows users are recommended to use [WSL](https://learn.microsoft.com/en-us/windows/wsl/install)
+or a Linux VM if they wish to use the REF.
+
+///
+
+Some metric providers can use their own conda environments.
+The REF can manage these for you,
+using a bundled version of [micromamba](https://github.com/mamba-org/micromamba-releases).
+
+The conda environments for the registered providers can be created with the following command:
+
+```bash
+ref --log-level=info providers create-env
+```
+
+A new environment will be created automatically for each conda-based metric provider when it is first used,
+if one does not already exist.
+This can cause issues if the environment is created on a node that doesn't have internet access,
+or if a race condition occurs when multiple processes try to create the environment at the same time.
+
+
+/// admonition | Note
+
+The PMP conda environment is not yet available for arm-based MacOS users,
+so the automatic installation process will fail.
+
+Arm-based MacOS users can use the following command to create the conda environment manually:
+
+```bash
+MAMBA_PLATFORM=osx-64 uv run ref providers create-env --provider pmp
+```
+
+///
+
+The created environments and their locations can be viewed using the command:
+
+```bash
+uv run ref providers list