Implement native SQL Alchemy MVT #2022

webb-ben · 2025-05-30T03:12:05Z

Overview

This PR extends upon the work in #1979 to implement the full query using SQL alchemy functions instead of SQL text blocks.

I would like to implement better testing as well for tiles.

Related Issue / discussion

Additional information

Dependency policy (RFC2)

I have ensured that this PR meets RFC2 requirements

Updates to public demo

I have ensured that breaking changes to the pygeoapi master demo server have been addressed
- https://github.com/geopython/demo.pygeoapi.io/blob/master/services/pygeoapi_master/local.config.yml

Contributions and licensing

(as per https://github.com/geopython/pygeoapi/blob/master/CONTRIBUTING.md#contributions-and-licensing)

I'd like to contribute [feature X|bugfix Y|docs|something else] to pygeoapi. I confirm that my contributions to pygeoapi will be compatible with the pygeoapi license guidelines at the time of contribution
I have already previously agreed to the pygeoapi Contributions and Licensing Guidelines

webb-ben · 2025-06-02T23:23:08Z

CI Failure unrelated to changes

webb-ben · 2025-06-05T20:47:18Z

@ThorodanBrom I can't request your review but hoping you can validate this?

ThorodanBrom · 2025-06-05T23:00:48Z

Sure, I'll test it out and add my review. Thanks! This looks much better that what we did.On 6 Jun 2025 2:17 am, Benjamin Webb ***@***.***> wrote:webb-ben left a comment (geopython/pygeoapi#2022) @ThorodanBrom I can't request your review but hoping you can validate this? —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>

Fix typo Rename base tile provider Fix Tile HTML highlighting Fix missing tile rename Update CRS conversion Change fallback for vector tile id Undo rename tile provider Reduce diff Cleanup mvt-postgres provider Re-add ST_CurveToLine geopython@97ba024

Fix Flake8

webb-ben · 2025-06-10T16:06:29Z

@tomkralidis @ThorodanBrom Added provider test and rebased. Should be ready for both of your reviews

doublebyte1

If I hit the tile metadata endpoint:

http://localhost:5000/collections/ne_110m_populated_places_simple/tiles/WorldCRS84Quad/metadata

I am getting this error:

500 -
Traceback (most recent call last):
File "/home/joana/git/tmp/pygeoapi/lib/python3.12/site-packages/flask/app.py", line 1536, in call
return self.wsgi_app(environ, start_response)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/joana/git/tmp/pygeoapi/lib/python3.12/site-packages/flask/app.py", line 1514, in wsgi_app
response = self.handle_exception(e)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/joana/git/tmp/pygeoapi/lib/python3.12/site-packages/flask_cors/extension.py", line 176, in wrapped_function
return cors_after_request(app.make_response(f(*args, **kwargs)))
^^^^^^^^^^^^^^^^^^^^
File "/home/joana/git/tmp/pygeoapi/lib/python3.12/site-packages/flask/app.py", line 1511, in wsgi_app
response = self.full_dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/joana/git/tmp/pygeoapi/lib/python3.12/site-packages/flask/app.py", line 919, in full_dispatch_request
rv = self.handle_user_exception(e)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/joana/git/tmp/pygeoapi/lib/python3.12/site-packages/flask_cors/extension.py", line 176, in wrapped_function
return cors_after_request(app.make_response(f(*args, **kwargs)))
^^^^^^^^^^^^^^^^^^^^
File "/home/joana/git/tmp/pygeoapi/lib/python3.12/site-packages/flask/app.py", line 917, in full_dispatch_request
rv = self.dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/joana/git/tmp/pygeoapi/lib/python3.12/site-packages/flask/app.py", line 902, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) # type: ignore[no-any-return]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/joana/git/tmp/pygeoapi/lib/python3.12/site-packages/pygeoapi/flask_app.py", line 357, in get_collection_tiles_metadata
return execute_from_flask(tiles_api.get_collection_tiles_metadata,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/joana/git/tmp/pygeoapi/lib/python3.12/site-packages/pygeoapi/flask_app.py", line 153, in execute_from_flask
headers, status, content = api_function(actual_api, api_request, *args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/joana/git/tmp/pygeoapi/lib/python3.12/site-packages/pygeoapi/api/tiles.py", line 317, in get_collection_tiles_metadata
tiles_metadata = p.get_metadata(
^^^^^^^^^^^^^^^
TypeError: BaseProvider.get_metadata() got an unexpected keyword argument 'dataset'

(you can reproduce my setup here: geopython/pygeoapi-examples#24)

ThorodanBrom · 2025-06-15T14:54:47Z

If I hit the tile metadata endpoint:

http://localhost:5000/collections/ne_110m_populated_places_simple/tiles/WorldCRS84Quad/metadata

I am getting this error:

500 -
Traceback (most recent call last):
File "/home/joana/git/tmp/pygeoapi/lib/python3.12/site-packages/flask/app.py", line 1536, in call
return self.wsgi_app(environ, start_response)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/joana/git/tmp/pygeoapi/lib/python3.12/site-packages/flask/app.py", line 1514, in wsgi_app
response = self.handle_exception(e)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/joana/git/tmp/pygeoapi/lib/python3.12/site-packages/flask_cors/extension.py", line 176, in wrapped_function
return cors_after_request(app.make_response(f(*args, **kwargs)))
^^^^^^^^^^^^^^^^^^^^
File "/home/joana/git/tmp/pygeoapi/lib/python3.12/site-packages/flask/app.py", line 1511, in wsgi_app
response = self.full_dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/joana/git/tmp/pygeoapi/lib/python3.12/site-packages/flask/app.py", line 919, in full_dispatch_request
rv = self.handle_user_exception(e)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/joana/git/tmp/pygeoapi/lib/python3.12/site-packages/flask_cors/extension.py", line 176, in wrapped_function
return cors_after_request(app.make_response(f(*args, **kwargs)))
^^^^^^^^^^^^^^^^^^^^
File "/home/joana/git/tmp/pygeoapi/lib/python3.12/site-packages/flask/app.py", line 917, in full_dispatch_request
rv = self.dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/joana/git/tmp/pygeoapi/lib/python3.12/site-packages/flask/app.py", line 902, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) # type: ignore[no-any-return]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/joana/git/tmp/pygeoapi/lib/python3.12/site-packages/pygeoapi/flask_app.py", line 357, in get_collection_tiles_metadata
return execute_from_flask(tiles_api.get_collection_tiles_metadata,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/joana/git/tmp/pygeoapi/lib/python3.12/site-packages/pygeoapi/flask_app.py", line 153, in execute_from_flask
headers, status, content = api_function(actual_api, api_request, *args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/joana/git/tmp/pygeoapi/lib/python3.12/site-packages/pygeoapi/api/tiles.py", line 317, in get_collection_tiles_metadata
tiles_metadata = p.get_metadata(
^^^^^^^^^^^^^^^
TypeError: BaseProvider.get_metadata() got an unexpected keyword argument 'dataset'

(you can reproduce my setup here: geopython/pygeoapi-examples#24)

I think this is because the get_metadata method from BaseProvider is getting called instead of the one from BaseMVTProvider. This may be due to PostgreSQLProvider becoming a class field.

Changing the constructor order to MVTPostgreSQLProvider (BaseMVTProvider, PostgreSQLProvider) seems to fix the issue

ThorodanBrom · 2025-06-15T15:45:17Z

A couple of bugs that I noticed:

Multiple features seem to get highlighted when clicked on the map on the pygeoapi UI. The current version does not seem to do this.
The fields/attributes are not returned as part of the MVT responses. This can't be tested using the pygeoapi UI, but on OpenLayers, only the layer name is returned.
Updated version:

Current version:

- Fix get_metadata function - Add unit test for get_metadata

- Fix MVT field selection - Add test for vector tile properties

Bump GitHub Postgis engine to >3

webb-ben · 2025-06-16T00:24:21Z

@doublebyte1 @ThorodanBrom Believe I have addressed both of your comments and added corresponding unit tests

webb-ben · 2025-06-16T00:53:00Z

I will note particularly long loading times to render the full table as MVT. There is not a strategy to drop features is there?

ThorodanBrom · 2025-06-16T12:20:12Z

I will note particularly long loading times to render the full table as MVT. There is not a strategy to drop features is there?

I know that tippecanoe has logic for dropping features, so you can maybe check how they do it. I don't think pg_tileserv or Martin support dropping features. You can consider using LIMIT at lower zoom levels, but you then need to ensure that the tile output looks OK at those lower zoom levels.

Also, I've noticed that Leaflet is a bit slow for visualizing MVTs (which is what pygeoapi UI uses). OpenLayers is a bit faster when I've used it.

ThorodanBrom · 2025-06-16T12:24:50Z

@webb-ben, I think WorldCRS84Quad tiles don't work properly. This is an issue from the original PR #1979. I've made a PR #2042 to fix it on master.

webb-ben · 2025-06-16T16:26:01Z

@webb-ben, I think WorldCRS84Quad tiles don't work properly. This is an issue from the original PR #1979. I've made a PR #2042 to fix it on master.

Can you confirm you have the correct configuration for storage_crs? The default storage_crs config option is http://www.opengis.net/def/crs/OGC/1.3/CRS84 but often Postgis geometry SRID is http://www.opengis.net/def/crs/EPSG/0/4326 with lat/lon switched.

With this configuration:

          - type: tile
            name: MVT-postgresql
            data:
                host: localhost
                dbname: test
                user: postgres
                password: postgres
                search_path: [osm, public]
            id_field: osm_id
            table: hotosm_bdi_waterways
            geom_field: foo_geom
            storage_crs: http://www.opengis.net/def/crs/EPSG/0/4326
            options:
                zoom:
                    min: 5
                    max: 15
            format:
                name: pbf
                mimetype: application/vnd.mapbox-vector-tile

I see:

webb-ben · 2025-06-16T16:36:56Z

I will note particularly long loading times to render the full table as MVT. There is not a strategy to drop features is there?

I know that tippecanoe has logic for dropping features, so you can maybe check how they do it. I don't think pg_tileserv or Martin support dropping features. You can consider using LIMIT at lower zoom levels, but you then need to ensure that the tile output looks OK at those lower zoom levels.

Also, I've noticed that Leaflet is a bit slow for visualizing MVTs (which is what pygeoapi UI uses). OpenLayers is a bit faster when I've used it.

@tomkralidis / @doublebyte1 / @ThorodanBrom I have a change (webb-ben@d32c6f9) that models the configuration for pygeoapi limit to control the maximum number of features in a single tile, ordered by the size of the feature. I will wait to include this as a separate PR unless directed otherwise. It could be valuable to combine spatial functions and property based ranking to better preserve only the most prominent features at low zoom.

          - type: tile
            name: MVT-postgresql
            ...
            options:
                limits:
                    max_items: 1000

ThorodanBrom · 2025-06-16T16:43:58Z

As far as I know, the pygeoapi UI cannot visualize WorldCRS84Quad tiles since it's based on leaflet (I think @doublebyte1 had mentioned this during the code sprint). So I think what you're seeing here is just WebMercatorQuad tiles again. I could be wrong here though.When I test WorldCRS84Quad, I usually use an OpenLayers map, and in OpenLayers, the existing implementation was incorrect. On 16 Jun 2025 9:56 pm, Benjamin Webb ***@***.***> wrote:webb-ben left a comment (geopython/pygeoapi#2022) @webb-ben, I think WorldCRS84Quad tiles don't work properly. This is an issue from the original PR #1979. I've made a PR #2042 to fix it on master. @webb-ben, I think WorldCRS84Quad tiles don't work properly. This is an issue from the original PR #1979. I've made a PR #2042 to fix it on master. Can you confirm you have the correct configuration for storage_crs? The default storage_crs config option is http://www.opengis.net/def/crs/OGC/1.3/CRS84 but often Postgis geometry SRID is http://www.opengis.net/def/crs/EPSG/0/4326 with lat/lon switched. With this configuration: - type: tile name: MVT-postgresql data: host: localhost dbname: test user: postgres password: postgres search_path: [osm, public] id_field: osm_id table: hotosm_bdi_waterways geom_field: foo_geom storage_crs: http://www.opengis.net/def/crs/EPSG/0/4326 options: zoom: min: 5 max: 15 format: name: pbf mimetype: application/vnd.mapbox-vector-tile I see: image.png (view on web) image.png (view on web) —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>

Prevent passing dict to SQL connection which throws an error because it cannot be cached

Align with WorldCRS84Quad spec using geopython#2042

webb-ben · 2025-06-17T01:25:44Z

@ThorodanBrom @doublebyte1 based on my understanding of WorldCRS84Quad, this PR should now implement the envelope logic in #2042

Add minimal support stretching EPSG3857 TileLayer to EPSG4326. Really should be a CRS84 based layer like: L.tileLayer.wms("https://gibs.earthdata.nasa.gov/wms/epsg4326/best/wms.cgi", { layers: "BlueMarble_ShadedRelief_Bathymetry", attribution: "NASA GIBS", }) .addTo(map);

ThorodanBrom · 2025-06-27T08:37:03Z

Commit 3384f69 now allows for WorldCRS84Quad tiles to be visible on the pygeoapi UI. The basemap does not align nicely with the tiles, but I think this is what the warning is for.

I'd suggest that OpenLayers be used for tile display instead of the current Leaflet implementation, because OpenLayers supports both WebMercatorQuad and WorldCRS84Quad MVTs seamlessly. It also supports raster tiles in both TMSs as well.

If this is something desirable, I can work on it.

tomkralidis · 2025-06-27T10:26:45Z

@ThorodanBrom the HTML/UI mapping library has a few considerations (example: CoverageJSON support, packaging/dependencies for Debian/Ubuntu/etc.). Suggest a separate issue for discussion.

webb-ben requested a review from doublebyte1 May 30, 2025 03:12

webb-ben force-pushed the mvt-postgres branch 7 times, most recently from c813098 to 26b7ec8 Compare June 2, 2025 22:41

webb-ben marked this pull request as ready for review June 2, 2025 23:23

webb-ben requested a review from a team June 2, 2025 23:23

webb-ben added OGC API - Tiles OGC API - Tiles ui User interface labels Jun 2, 2025

webb-ben force-pushed the mvt-postgres branch from 46ab502 to 4d1a6d3 Compare June 2, 2025 23:34

tomkralidis self-requested a review June 3, 2025 10:57

tomkralidis added this to the 0.21.0 milestone Jun 3, 2025

webb-ben force-pushed the mvt-postgres branch 3 times, most recently from 1ab9ae2 to 690c0ed Compare June 10, 2025 15:54

webb-ben added 4 commits June 10, 2025 11:54

Add ST_CurveToLine for output geom

08ada27

Set id field for vector files

b3a9e9a

Add MVT Postgres Provider test

2fb298d

Fix Flake8

webb-ben force-pushed the mvt-postgres branch from 690c0ed to 2fb298d Compare June 10, 2025 15:54

Fix indentation in MVT-Postgres docs

6b7b3c8

doublebyte1 reviewed Jun 15, 2025

View reviewed changes

webb-ben added 6 commits June 15, 2025 18:40

Fix get_metadata()

23f35a7

- Fix get_metadata function - Add unit test for get_metadata

Fix MVT feature properties

8e0b7f3

- Fix MVT field selection - Add test for vector tile properties

Fix flake8

5eb616c

Add storage_crs information to docs

740180c

Add test_postgresql_mvt_provider to GitHub CI

615288d

Bump GitHub Action Postgis version

ddb1ba2

Bump GitHub Postgis engine to >3

webb-ben requested a review from doublebyte1 June 16, 2025 00:24

webb-ben added 3 commits June 16, 2025 21:02

Filter SQL connection options

f71a63f

Prevent passing dict to SQL connection which throws an error because it cannot be cached

Use geoalchemy2 functions

75cac23

Amend WorldCRS84Quad Tile bbox

8b7d44f

Align with WorldCRS84Quad spec using geopython#2042

webb-ben mentioned this pull request Jun 24, 2025

Bug-Fix : Fix tile fetch code for MVT Postgres for WorldCRS84Quad TMS #2042

Closed

5 tasks

webb-ben mentioned this pull request Jun 27, 2025

Investigate better CRS handling in HTML templates #2050

Open

tomkralidis merged commit 350aec1 into geopython:master Jun 28, 2025
4 checks passed

webb-ben deleted the mvt-postgres branch June 30, 2025 12:31

webb-ben mentioned this pull request Jul 3, 2025

search_path option for mvt_postgresql #2025

Open

Uh oh!

Implement native SQL Alchemy MVT #2022

Implement native SQL Alchemy MVT #2022

Uh oh!

Conversation

webb-ben commented May 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Related Issue / discussion

Additional information

Dependency policy (RFC2)

Updates to public demo

Contributions and licensing

Uh oh!

webb-ben commented Jun 2, 2025

Uh oh!

webb-ben commented Jun 5, 2025

Uh oh!

ThorodanBrom commented Jun 5, 2025 via email

Uh oh!

webb-ben commented Jun 10, 2025

Uh oh!

doublebyte1 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ThorodanBrom commented Jun 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ThorodanBrom commented Jun 15, 2025

Uh oh!

webb-ben commented Jun 16, 2025

Uh oh!

webb-ben commented Jun 16, 2025

Uh oh!

ThorodanBrom commented Jun 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ThorodanBrom commented Jun 16, 2025

Uh oh!

webb-ben commented Jun 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

webb-ben commented Jun 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ThorodanBrom commented Jun 16, 2025 via email

Uh oh!

webb-ben commented Jun 17, 2025

Uh oh!

ThorodanBrom commented Jun 27, 2025

Uh oh!

tomkralidis commented Jun 27, 2025

Uh oh!

Uh oh!

Uh oh!

webb-ben commented May 30, 2025 •

edited

Loading

doublebyte1 left a comment •

edited

Loading

ThorodanBrom commented Jun 15, 2025 •

edited

Loading

ThorodanBrom commented Jun 16, 2025 •

edited

Loading

webb-ben commented Jun 16, 2025 •

edited

Loading

webb-ben commented Jun 16, 2025 •

edited

Loading