Skip to content
This repository was archived by the owner on Jul 11, 2025. It is now read-only.

Improve README with clear structure and added quick start guide #54

Merged
merged 1 commit into from
Mar 29, 2025
Merged

Improve README with clear structure and added quick start guide #54

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
190 changes: 147 additions & 43 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,48 @@ API for supporting reef suitability assessments.

[![Documentation](https://img.shields.io/badge/docs-dev-blue)](https://open-aims.github.io/ReefGuideAPI.jl/dev/)


## Table of Contents

- [Quick Start](#quick-start)
- [Setup](#setup)
- [Configuration](#configuration)
- [JWT Authentication](#jwt-auth-configuration)
- [API Usage](#api-usage)
- [Development](#development-setup)
- [Performance Notes](#performance-notes)
- [Reef Edge Alignment](#reef-edge-alignment-for-site-searching)
- [Docker Usage](#docker-build-and-run)
- [Dockerfile Configuration](#dockerfile-configuration)
- [Publishing Releases](#publish-release)
- [Data Requirements](#mounting-files-and-required-data)
- [Building and Running](#building-and-running)
- [Troubleshooting](#troubleshooting)

## Quick Start

Start the API server with a configuration file:

```julia
using ReefGuideAPI

# To enable debug messages:
# ENV["JULIA_DEBUG"] = "ReefGuideAPI"

# If multiple threads are available, a parallel server will be spun up
ReefGuideAPI.start_server(".config.toml")
```

### Using Docker

```bash
# Run with Docker
docker run -p 8000:8000 -v ./data:/data/reefguide reefguide

# Or with Docker Compose
docker compose up reefguide-src
```

## Setup

Initialize the project the usual way:
Expand All @@ -29,9 +71,9 @@ TILE_SIZE = "256" # Optional, tile block size to use (defaults to 256)

By convention, this file is named `.config.toml` (note the leading `.`).

## JWT Auth configuration
## JWT Auth Configuration

The API can be additionally configured to expect a valid JWT in the `Authorization: Bearer <token>` header format.
The API can be additionally configured to expect a valid JWT in the `Authorisation: Bearer <token>` header format.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

😭 my z!

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I prefer US spelling for code and documentation these days. I'm sorry.


Add the following to `.config.toml`:

Expand All @@ -42,50 +84,41 @@ JWT_ENABLED = true
# Which iss to validate for the JWTs?
JWT_ISS = "https://issuer.com"
# WKT JWKS endpoint where public key can be retrieved
WKT_ENDPOINT = "https://https://issuer.com/api/.well-known/jwks.json"
WKT_ENDPOINT = "https://issuer.com/api/.well-known/jwks.json"
```

Pay attention to the issuer and wkt endpoints. The first should exactly match the expected JWT issuer claim. The second should be web-resolvable and return a WKT JSON which provides the public key.

### Auth TODOs
### Authentication Notes

- ensure health check route is not authorised
- When JWT authentication is enabled, most API routes require a valid token
- The health check endpoint (`/health`) remains accessible without authentication
- Tokens must be included in the Authorisation header format: `Authorization: Bearer <token>`
- For testing purposes, you can disable authentication by setting `JWT_ENABLED = false`

## Quickstart
## API Usage

```julia
using ReefGuideAPI

# To enable debug messages:
# ENV["JULIA_DEBUG"] = "ReefGuideAPI"

# If multiple threads are available, a parallel server will be spun up
ReefGuideAPI.start_server(".config.toml")
```

In its current state, the main page displays a simple form for dev/testing purposes.

## Dynamic COG generation
### Dynamic COG Generation

Example URL:

```code
```
http://127.0.0.1:8000/assess/Cairns-Cooktown/slopes?Depth=-9.0:0.0&Slope=0.0:40.0
```

## Simple Slippy Tiles
### Simple Slippy Tiles

Example URL:

```code
```
http://127.0.0.1:8000/tile/8/231/139?region=Cairns-Cooktown&rtype=slopes&Depth=-9.0:0.0&Slope=0.0:40.0&Rugosity=0.0:3.0
```

## Development setup
## Development Setup

The steps below assumes you are in the project root.

Create a sandbox enviroment:
Create a sandbox environment:

```bash
$ mkdir sandbox
Expand All @@ -111,14 +144,14 @@ Note that the server now caches the initially loaded spatial data in between ser
launches to reduce downtime. It will be necessary to restart the Julia session to reload
spatial data.

## Performance notes
## Performance Notes

The config setting `COG_THREADS` controls how many threads should be requested when writing
out COGs. Ideally this will be set to at least 2 (preferably 4).
Higher values do seem to reduce write times but with diminishing returns (tested up to 8).
Locally, write times with four threads configured range from 10 to 15 seconds.

## Reef edge alignment for site searching
## Reef Edge Alignment for Site Searching

`identify_edge_aligned_sites()` can be used to identify potential sites that only align with
the nearest reef edge (or specified rotations away from this angle).
Expand All @@ -134,64 +167,135 @@ The following processing is required before use:
- The target region name should be specified in GBRMPA format.
- E.g. "Townsville/Whitsunday Management Area" rather than "Townsville-Whitsunday".

### Parquet assessment additional setup
### Parquet Assessment Additional Setup

- A parquet GeoDataFrame must be loaded and filtered for unsuitable pixels based on user criteria thresholds using a Dict and `within_thresholds()`.
- `lons` and `lats` columns (FLoat64) must be added to the GeoDataFrame.
- E.g. `valid_pixels.lons = first.(GI.coordinates.(valid_pixels.geometry))`
The column used for masking should be the same as the column specified as geometry_col in
`identify_edge_aligned_sites` (default = `:geometry`).

## Docker build and run
## Docker Build and Run

The ReefGuideAPI.jl package has an associated `Dockerfile` and build/publish process. This means you can run an instance of the ReefGuideAPI.jl package without needing to compile/build it with a local `Julia` installation. You will be able to view the latest published versions of the Docker image on the repository packages page.

### Mounting files and required data
### Dockerfile Configuration

As mentioned above, the `ReefGuideAPI.jl` package currently requires
The Julia version is specified by the `JULIA_VERSION` arg. The full version is specified
to maintain build stability, but should be bumped to the latest version of Julia when a
release is published.

#### A Note About MKL_jll

Due to how Julia (particularly v1.11) handles precompilation, it significantly reduces the build time by explicitly installing MKL_jll before installing any explicit project dependencies.

For this reason, the Dockerfile extracts the MKL_jll version from the Manifest file using Pkg.dependency(), precompiles this in an anonymous project, then compiles the main dependencies. This cuts the build time from around 15 minutes down to around 6-7.

### Publish Release

To publish a new version of the Docker image:

1. Bump version in Project.toml and [PublishDockerImage.yml](../../.github/workflows/PublishDockerImage.yml)
2. Create PR, merge to `main` branch
3. Publish Release on GitHub (this triggers the PublishDockerImage workflow)

### Mounting Files and Required Data

The `ReefGuideAPI.jl` package requires:

- a `config.toml` file and
- a set of input data files

Please include these in a folder called `data` in your working directory.

When running the below commands, it is assumed you have `data` available locally with the required files.
When running the commands below, it is assumed you have `data` available locally with the required files.

**Note**: Due to how Docker excludes `.` files, we have named the config file `config.toml` in the data folder (not `.config.toml`). This is required to launch the server.

**Note**: Due to how Docker excludes `.` files, we have named the config file `config.toml` in the data folder. This is required to launch the server.
### Building and Running

### To build from src files using Docker
#### Build from Source Files

Using Docker:
```bash
docker build . --target reefguide-src -t reefguide
```

### To build from src files using Docker Compose

Using Docker Compose:
```bash
docker compose build reefguide-src
# Or to build and run in one command:
docker compose up --build reefguide-src
```

### To run with mounted files (launch server) using Docker
#### Run Server with Mounted Files

Using Docker:
```bash
docker run -p 8000:8000 -v ./data:/data/reefguide reefguide
```

### To run with mounted files (launch server) using Docker Compose

Using Docker Compose:
```bash
docker compose up reefguide-src
```

### To run with mounted files (interactive shell) using Docker
#### Run Interactive Shell with Mounted Files

This will start a Julia shell where `ReefGuideAPI` is compiled and ready for use:

```bash
docker run --rm --interactive --entrypoint="julia" --tty -v ./data:/data/reefguide reefguide
```

This will start a Julia shell where `ReefGuideAPI` is compiled and ready for use e.g.
Then in the Julia REPL:

```julia
using ReefGuideAPI
ReefGuideAPI.start_server("/data/reefguide/config.toml")
```

```julia
docker run --rm --interactive --entrypoint="julia" --tty -v ./data:/data/reefguide reefguide
```
## Troubleshooting

### Common Issues

#### Configuration Problems

**Issue**: Server fails to start with configuration error
- **Solution**: Double-check that your config file has the correct path format for your OS (Windows uses backslashes, Unix uses forward slashes)
- **Solution**: Ensure all required directories specified in the config actually exist on your system

**Issue**: Data not found
- **Solution**: Verify the `PREPPED_DATA_DIR` path is correct and contains the required MPA dataset files
- **Solution**: When using Docker, check that the volume mount path (`-v ./data:/data/reefguide`) is correct and the `data` directory contains your files

#### Docker-Specific Issues

**Issue**: Docker container exits immediately after starting
- **Solution**: Check Docker logs with `docker logs <container-id>` to see the specific error
- **Solution**: Verify that your `config.toml` is named correctly (not `.config.toml`) in the data directory

**Issue**: "Permission denied" errors when accessing data directory
- **Solution**: Check file permissions on your data directory and ensure the Docker user has read/write access

#### Performance Issues

**Issue**: Slow COG generation
- **Solution**: Increase the `COG_THREADS` value in your config (recommend 2-4)
- **Solution**: Ensure the server has sufficient memory for processing (minimum 4GB recommended)


#### Authentication Issues

**Issue**: JWT token rejected as invalid
- **Solution**: Verify the token's issuer (`iss` claim) matches exactly what's in your config
- **Solution**: Check that the token is not expired
- **Solution**: Ensure the `WKT_ENDPOINT` is accessible from the server

### Getting Help

If you encounter issues not covered here, please:

1. Check the Julia REPL output for specific error messages
2. Enable debug logs with `ENV["JULIA_DEBUG"] = "ReefGuideAPI"`
3. Open an issue on the GitHub repository with a detailed description of the problem