NMBGMRCGDMS POCs Pros/Cons

[jirhiker]

New Mexico Bureau of Geology and Mineral Resources Centralized Geospatial Data Management System

NMSqlalchemyPOC (aka NMSampleLocations).

pros.

1. Fastapi: familiar to devs. Pydantic is sweet. modern. great documentation
2. API first.
3. "full" control of design

cons

1. No out-of-the-box admin interface.
2. build from scratch

GeoDjango

pros

1. batteries included

cons

1. opinionated
2. ORM is farther removed from SQL than Sqlalchemy

Thought

The admin interface in Django is useful for development. Do we think it would be used directly by users, or will they only interface with Data Manager? The users commonly complain about one-to-one UI-to-table applications. Does Django replace Data Manager for Data Management? and Data Manager becomes Data Viewer?

[jacob-a-brown]

I don't know if this is a pro or a con, but the directory structure of GeoDjango is rigid due to it being opinionated. Because of that, new developers would, in theory, be familiar with project's directory structure (if they are already familiar with Django projects). FastAPI, on the other hand, really allows any sort of directory structure, so it would take more time to onboard new developers. That being said, we could (and should) document and enforce a particular type of directory structure for a FastAPI project.

[jacob-a-brown]

I also think that the admin panel - FastAPI or Django - may be able to replace the single table viewers and editors envisioned in the current state of DataManager. Using the admin panels would save us a lot of time since those single-table viewers and editors would, essentially, already be made for us. A hudrle would be getting the style consistent between DataManger (which could be used for complex views and edits, such as a well viewer that shows location info, well info, and contact/owner info) and the admin page (which would be used for single tables).

[jirhiker]

Refine.dev framework makes building the single table views quite easy. Its very boilerplate and requires pretty simple API endpoints

[jacob-a-brown]

After doing some development in Django here are some initial thoughts (more to come after I continue work in the repo). This may be slightly biased because I'm used to FastAPI and so that's the lens through which I'm viewing the pros and cons:

Pros

• Django Ninja enables me to set up bespoke endpoints and functions very much like FastAPI, so you can essentially make the same API in the same manner
• Handles both swagger and Redoc
• Models are easy to define and handle
• Migrations are very easy to handle and use
• Admin panel is clean and easy to set up and use

Cons

• The directory structure is kind of confusing
• I don't really like that everything needs to be run through manage.py (like the tests). I find that this obfuscates many of the functionalities. It is easily findable through the docs and via AI, but it's still kind of confusing. I kind of like that FastAPI uses different libraries that I can just call from the command line (e.g. pytest)
• I don't like using relative imports. I find that they are fickle and often break. I prefer for the application to be run from the root directory and for every directory to have an __init__.py file in it so that one can import in a simpler fashion (e.g. from models.locations import locations instead of from ..models.locations import locations if that is the relative path to models)
• Django Ninja does not have as large a user base as FastAPI, which may make support more difficult

[jirhiker]

@jacob-a-brown what are your thoughts on Django orm vs sqlachemy?

[jacob-a-brown]

I haven't used DjangoORM much yet. I've written the tests in the SQL Alchemy POC and will start to implement the endpoints for them more robustly. I'll note down advantages/disadvantages of them after I spend more time in development.

[jacob-a-brown]

Some other pros of DjangoNinja are that it has built-in pagination and versioning, whereas those are done by third-party FastAPI libraries that are not well-maintained (less of an issue with pagination as that's easy to implement ourselves).

[jacob-a-brown]

Another con is that the auto-generated docs do not show the schema(s)

[jacob-a-brown]

Some more thoughts...

It seems to me that the biggest advantage of Django is the admin panel and all of its extensions, whereas the biggest advantage of FastAPI is its speed and "API-first" ethos. So, at least as I understand it at the moment, a big question is: do we want to prioritize Django and its admin panel (and extensions), or API development with FastAPI?

Do others have this sense, too, or am I off with this intuition? I still need to work with Django ORM so don't yet have a good feel for it. That may change my thoughts as I use it.

[jacob-a-brown]

It looks like there are far fewer FastAPI extensions, some of which are no longer maintained. FastAPI Versionizer, for example, is no longer maintained. We could inherit and maintain that library, but do we want to take on this responsibility? It could add to our developer and team cred, but would also take time away from development and furtherance of the Bureau's and WDI's goals.

[jirhiker]

I agree with your synopsis of Django vs FastAPI.

Django has a lot of extensions, how many of them are still maintained?

"FastAPI is a micro-framework focused on APIs, while Django is a full-stack framework for complete web applications." I feel like since we are planning on building the UI with React, FastAPI is a better choice.

[chasetmartin]

A few notes I have after using Django during this sprint:
Pros:

• models and migrations are extremely straightforward and easy to use. Even when it comes to setting up a TimescaleDB hypertable that Django doesn't know anything about, you can just create a migration file and then add the create table sql yourself (which runs once and creates the hypertable) I was worried about this functionality, but it seems to work just fine:

        migrations.RunSQL(
            """
            CREATE TABLE samplelocations_groundwaterlevel (
                well_id INT NOT NULL,
                level FLOAT NOT NULL,
                timestamp TIMESTAMPTZ NOT NULL,
                PRIMARY KEY (well_id, timestamp),
                FOREIGN KEY (well_id) REFERENCES samplelocations_well(id),
            );
            SELECT create_hypertable('samplelocations_groundwaterlevel', 'timestamp', 'well_id', 2);
            """,
            reverse_sql="""
            DROP TABLE samplelocations_groundwaterlevel;
            """
        )

• The batteries included admin panel is extremely useful for basic CRUD operations, and a well-maintained theme like the Unfold admin theme adds some useful helpers and organizational features that make it even more useful. It's never going to do super advanced queries, but I think it's worth discussing just how much of a normal "data management" workflow the admin panel can take care of. You can even develop custom forms within the admin panel.
• The opinionated nature and structure of Django makes it easy to understand from project to project - if you've worked with Django before you immediately know what's going on
• Templating can be very powerful if done properly (just see the Unfold admin panel as an example). Obviously has it's limitations.
• Django Ninja routing behavior seems to provide a somewhat similar developer experience to FastAPI.
• Django documentation is plentiful.

CONS:

• GeoDjango cannot run on a base python environments provided by many deployment offerings because of GDAL - customs containers are necessary it seems.
• The opinionated nature can be combersome you're trying to build the entire stack within Django (admin panel, front end with templates, api, migrations). It can feel 'bulky' to work with, if that makes sense.
• Templates will always be server based, so the admin panel is never going to have the responsiveness of a React SPA.

Just like Jacob said: do we value the batteries included in the Django admin panel vs Fast admin? Or do we focus on the api first approach? How many of the data management issues can be solved by the out of the box implementation of DjangoORM with the admin panel? 50%? 90%? How many users would you give access to the admin panel? The admin panel can't do everything and isn't designed to, so at what point does it become a burden to maintain custom admin features if you're also still building the React-based Data Manager?

[jacob-a-brown]

I like your summaries. I think the pros/cons you enumerated, as well as the questions at the end, give us some good discussion points and foot for thought. Definitely a lot to consider as we hone in on our framework and tech stack.

I'm finishing up some model development for Kelsey as she goes on vacation. One of the struggles is getting the PointField to render and populate correctly from the admin panel. It seems possible with widgets, but that requires more in-depth research and bespoke implementation. I'm just calling out that field because a Location's geometry (point?) is fundamental to all place-based work.

[chasetmartin]

@jacob-a-brown Yeah that's a limitation of Unfold widgets. I have a branch that uses Leaflet for the map that works. I'm guessing we could integrate mapbox to do the same thing.

But it would also help to figure out a way to just input the lat / long, since clicking on a map every time is annoying. So yes - this would require some bespoke admin work.

[jacob-a-brown]

CoPilot gave me a working solution (I haven't merged pre-production with this branch yet so not sure how it would interact with the map):

from django.contrib import admin
from django import forms
from django.contrib.gis.admin import GISModelAdmin
from django.contrib.gis.geos import Point

from samplelocations.models import Location

class LocationForm(forms.ModelForm):
    x = forms.FloatField(label="Longitude")
    y = forms.FloatField(label="Latitude")
    z = forms.FloatField(label="Elevation")


    class Meta:
        model = Location
        fields = ["x", "y", "z"]

    def save(self, commit=True):
        self.instance.coordinate = Point(self.cleaned_data["x"], self.cleaned_data["y"], self.cleaned_data["z"], srid=4326)
        return super().save(commit=commit)

@admin.register(Location)
class LocationAdmin(GISModelAdmin):
    form = LocationForm
    list_display = ("location_id", "coordinate", "date_created")

[chasetmartin]

Yeah the map not displaying is a css issue hiding it, so with some custom css I got it to show:

The custom form field you show above would work for custom lat / long / elevation fields.

[jirhiker]

Fastapi vs Django benchmarks

TL;DR. Fastapi is faster than Django

https://www.techempower.com/benchmarks/#section=test&runid=7464e520-0dc2-473d-bd34-dbdfd7e85911&hw=ph&test=db&l=zik0zj-cn2&f=zik0zj-zik0zj-zhb2tb-zih7un-zik0zj-zik0zj-zik0zj-zik0zj-zik0zj-zik0zj-zik0zj-zik0zj-zik0zj-zik0zj-zik0zj-3j

[jacob-a-brown]

A ding against Alembic is that all models must be at least imported into another file for them to be recognized by SQL Alchemy's base and therefore recognized by Alembic. This is currently handled in db/__init__.py as

# import all models from db package so that Alembic can discover them
from db.asset import *
from db.base import *
from db.collabnet import *
from db.contact import *
from db.geochronology import *
from db.geothermal import *
from db.group import *
from db.lexicon import *
from db.location import *
from db.observation import *
from db.publication import *
from db.sample import *
from db.sensor import *
from db.series import *
from db.thing import *
from db.base import Base

This means that we must manually keep track of all relevant models/modules. This can, however, be mitigated by putting all of the models into a single file (e.g. models.py) and then in db/__init__.py call from db.models import *. That may make such a massive file a somewhat more difficult to navigate and understand, but would have the advantage of not needing to track every module that contains models (particularly if new ones are made).