Add provider for MVT tile generation from Postgres #1979

ThorodanBrom · 2025-03-27T14:15:42Z

Overview

This PR adds MVT tile generation from Postgres/PostGIS tables natively.

Related Issue / discussion

This is related to issue #1978

Additional information

Dependency policy (RFC2)

I have ensured that this PR meets RFC2 requirements

Updates to public demo

I have ensured that breaking changes to the pygeoapi master demo server have been addressed
- https://github.com/geopython/demo.pygeoapi.io/blob/master/services/pygeoapi_master/local.config.yml

Contributions and licensing

(as per https://github.com/geopython/pygeoapi/blob/master/CONTRIBUTING.md#contributions-and-licensing)

I'd like to contribute this feature to pygeoapi. I confirm that my contributions to pygeoapi will be compatible with the pygeoapi license guidelines at the time of contribution
I have already previously agreed to the pygeoapi Contributions and Licensing Guidelines

doublebyte1

Thank You @ThorodanBrom ! Would you be able to add a Postgres Provider docker example at pygeoapi-examples? https://github.com/geopython/pygeoapi-examples/tree/main/docker

doublebyte1 · 2025-04-19T11:06:02Z

pygeoapi/provider/mvt_postgres.py

+            query = text("""
+                WITH                                                                  
+                    bounds AS (
+                        SELECT ST_TileEnvelope(:z, :y, :x, ST_MakeEnvelope(-180, -90, 180, 90, 4326)) AS boundgeom


Why the tile envelope here is based on the extent of the crs, instead of the extent of the geometry (like in the case of WebMercator)?

ST_TileEnvelope by default is configured to EPSG:3857. In order to use it for other CRSs, the docs mention that you can add an extra parameter of a geometry that has an extent of the 0th zoom level. There's an example of setting it for EPSG:4326, we just used that here. (hope that this answered your question, if not, please let me know)

We can probably use the WKB representation there so that MakeEnvelope isn't calculated for every tile request.

I agree, I think the WKB representation would make sense here!

Addressed in 97ba024. I used EWKT representation instead of WKB as the the WKB hex string was too long

doublebyte1 · 2025-04-19T11:08:42Z

pygeoapi/provider/mvt_postgres.py

+                        SELECT ST_TileEnvelope(:z, :y, :x) AS boundgeom
+                    ),
+                    mvtgeom AS (
+                        SELECT ST_AsMVTGeom(ST_Transform({geom}, 3857), bounds.boundgeom) AS geom {fields}


I think converting curves to lines (ST_CurveToLine), could speed up queries in the case of more complex geometries.

OK, I will try this out, it looks like Martin also uses it.

So by using ST_CurveToLine, will this provider also be able to support CIRCULARSTRING-like data types (that is what I understood from reading the documentation)? Or is it just going to be used for query performance?

I think simplification, in general, yields performance; and in the case of tiles, we do not need the geometries to be so detailed.

Addressed in 97ba024

doublebyte1 · 2025-04-19T11:31:17Z

Please check the errors when building the documentation.
If possible, please also add a test for this provider. You can check this one for inspiration: https://github.com/geopython/pygeoapi/blob/master/tests/test_postgresql_provider.py

ThorodanBrom · 2025-04-21T12:19:02Z

Thanks for the review,

I'll work on adding an example provider for this in pygeoapi-examples as well as writing test cases
I'll check out the errors in the documentation and flake8
The PR needs a few updates and also needs to be rebased as well to accomodate the MVT tile fixes, is it OK if I rebase and force-push or will that break any of the CI flows?

doublebyte1 · 2025-04-26T12:49:53Z

Thanks for the review,

I'll work on adding an example provider for this in pygeoapi-examples as well as writing test cases

I'll check out the errors in the documentation and flake8

The PR needs a few updates and also needs to be rebased as well to accomodate the MVT tile fixes, is it OK if I rebase and force-push or will that break any of the CI flows?

Thank You @ThorodanBrom ! I think it is ok to rebase, but I am pinging @tomkralidis here, just in case.

tomkralidis · 2025-04-27T12:54:42Z

+1 to rebase, given it will start CI again fresh.

- `/tiles` and `/tiles/{TMS}` APIs are working

- Supports properties - Supports both WebMercatorQuad, WorldCRS84Quad Co-authored-by: PRAJWAL S <praju18299@gmail.com> Co-authored-by: Tanvi Prasad <tanvi.prasad@datakaveri.org>

- Fixed flake8 issues - Corrected tile indices in `ST_TileEnvelope` - Previous indices worked when tile URL was `/z/x/y`. Current ones work for `/z/y/x`, which is OGC-compliant - Used `ST_CurveToLine` when querying for features during tile generation to potentially speed up queries - Martin also uses it (https://maplibre.org/martin/sources-pg-functions.html#simple-function) - Used EWKT representation of `ST_MakeEnvelope(-180,-90,180,90,4326)` for WorldCRS84Quad tiles

doublebyte1 · 2025-05-08T15:21:04Z

docs/source/data-publishing/ogcapi-tiles.rst

+           table: hotosm_bdi_waterways
+           geom_field: foo_geom
+           options:
+           zoom:


@ThorodanBrom I think that the zoom should be placed under options

doublebyte1

Please fix the format of the example in the documentation, and I think we are good to go.

You can also click on the check-boxes that say "Updates to public demo", as this provider is not used in the configuration file of the demo.

Thank You!

ThorodanBrom · 2025-05-08T16:53:27Z

Thanks for the review @doublebyte1. I'm working on the example in pygeoapi-examples, but I'm not really sure on how to write tests for this provider. There aren't any tests for other MVT providers, so I don't have a reference. I can think of the following conditions to test:

if the provider is able to create tiles for the configured table and configured zoom levels for WebMercatorQuad and WorldCRS84Quad
if the tiles returned are valid MVT files + a 200 OK
- I don't know how to test for a 'valid' MVT, is there a library or something that can parse and validate the API output
if tiles requested for an invalid index or where data does not exist returns a 204 No Content

Do let me know if this sounds OK.

doublebyte1 · 2025-05-09T12:40:59Z

Thanks for the review @doublebyte1. I'm working on the example in pygeoapi-examples, but I'm not really sure on how to write tests for this provider. There aren't any tests for other MVT providers, so I don't have a reference. I can think of the following conditions to test:

if the provider is able to create tiles for the configured table and configured zoom levels for WebMercatorQuad and WorldCRS84Quad

if the tiles returned are valid MVT files + a 200 OK

I don't know how to test for a 'valid' MVT, is there a library or something that can parse and validate the API output

if tiles requested for an invalid index or where data does not exist returns a 204 No Content

Do let me know if this sounds OK.

You are right in that there are no tests for other MVT providers. Let's keep it like this, for now. Thank You for the availability!

tomkralidis

Great work here! Minor change requests on naming consistency.

tomkralidis · 2025-05-11T13:44:35Z

docs/source/data-publishing/ogcapi-tiles.rst

@@ -22,6 +22,7 @@ pygeoapi core tile providers are listed below, along with supported features.
   `MVT-elastic`_,✅,✅,✅,❌,❌,✅
   `MVT-proxy`_,❓,❓,❓,❓,❌,✅
   `WMTSFacade`_,✅,❌,✅,✅,✅,❌
+   `MVT-postgres`_,✅,✅,✅,✅,❌,✅


Rename to MVT-postgresql

tomkralidis · 2025-05-11T13:44:57Z

docs/source/data-publishing/ogcapi-tiles.rst

+.. note::
+   Requires Python packages sqlalchemy, geoalchemy2 and psycopg2-binary
+
+Must have PostGIS installed with protobuf-c support. 


Turn into an additional note

tomkralidis · 2025-05-11T13:45:23Z

docs/source/data-publishing/ogcapi-tiles.rst

+.. note::
+   Geometry must be using EPSG:4326
+
+This provider gives support to serving tiles generated using `Postgres <https://www.postgresql.org/>`_ with `PostGIS <https://postgis.net/>`_.


s/Postgres/PostgreSQL/g

tomkralidis · 2025-05-11T13:45:52Z

docs/source/data-publishing/ogcapi-tiles.rst

+
+   providers:
+       - type: tile
+           name: MVT-postgres


MVT-postgresql

tomkralidis · 2025-05-11T13:46:49Z

pygeoapi/plugin.py

@@ -53,6 +53,7 @@
        'MVT-tippecanoe': 'pygeoapi.provider.mvt_tippecanoe.MVTTippecanoeProvider',  # noqa: E501
        'MVT-elastic': 'pygeoapi.provider.mvt_elastic.MVTElasticProvider',
        'MVT-proxy': 'pygeoapi.provider.mvt_proxy.MVTProxyProvider',
+        'MVT-postgres': 'pygeoapi.provider.mvt_postgres.MVTPostgresProvider',


'MVT-postgresql': 'pygeoapi.provider.mvt_postgresql.MVTPostgreSQLProvider',

tomkralidis · 2025-05-11T13:47:47Z

pygeoapi/provider/mvt_postgres.py

Rename to pygeoapi/provider/mvt_postgresql.py

tomkralidis · 2025-05-11T13:48:55Z

pygeoapi/provider/mvt_postgres.py

@@ -0,0 +1,246 @@
+# =================================================================
+#
+# Authors:


Please add yourself as author. Example in https://github.com/geopython/pygeoapi/blob/master/setup.py#L3

tomkralidis · 2025-05-11T13:49:12Z

pygeoapi/provider/mvt_postgres.py

+#
+# Authors:
+#
+# Copyright (c)


Please add copyright. Example in https://github.com/geopython/pygeoapi/blob/master/setup.py#L5

tomkralidis · 2025-05-11T13:51:17Z

pygeoapi/provider/mvt_postgres.py

+#
+# =================================================================
+
+import logging


Order imports by standard library, 3rd party packages, then local imports, alphabetically, separated by blank line.

from copy import deepcopy import logging from sqlalchemy.sql import text from pygeoapi.models.provider.base import ( TileSetMetadata, TileMatrixSetEnum, LinkType) from pygeoapi.provider.base import ProviderConnectionError from pygeoapi.provider.base_mvt import BaseMVTProvider from pygeoapi.provider.postgresql import PostgreSQLProvider from pygeoapi.util import url_join

tomkralidis · 2025-05-11T13:51:34Z

pygeoapi/provider/mvt_postgres.py

+LOGGER = logging.getLogger(__name__)
+
+
+class MVTPostgresProvider(BaseMVTProvider):


Rename to MVTPostgreSQLProvider

- Updated all necessary files to change postgres -> postgresql related to the plugin - Updated documentation - Added authors to plugin file and organized imports

…S, return ProviderTileNotFound error - Otherwise error was thrown from PostgreSQL `ST_MakeTileEnvelope` function

ThorodanBrom · 2025-05-19T16:50:52Z

Hello @tomkralidis, thanks for the review. I think I've addressed all your comments in 9a320cc. (I added a small bug fix in a034d15 as well). Please let me know if there are any other required changes.

photonbit · 2025-05-21T11:29:51Z

docs/source/data-publishing/ogcapi-tiles.rst

+   Must have PostGIS installed with protobuf-c support 
+
+.. note::
+   Geometry must be using EPSG:4326


I understand that this note is outdated for the postgresql provider. Is the note valid for this case?

ThorodanBrom changed the title ~~Add mvt postgres provider~~ Add provider for MVT tile generation from Postgres Mar 27, 2025

ThorodanBrom mentioned this pull request Mar 27, 2025

Added documentation for the MVTPostgresProvider class #1980

Closed

5 tasks

tomkralidis requested review from tomkralidis and doublebyte1 March 30, 2025 12:10

tomkralidis added this to the 0.21.0 milestone Mar 30, 2025

tomkralidis added the OGC API - Tiles OGC API - Tiles label Mar 30, 2025

doublebyte1 reviewed Apr 19, 2025

View reviewed changes

ThorodanBrom and others added 4 commits May 2, 2025 22:32

Initial commit for adding MVT Postgres provider

cc4bba3

- `/tiles` and `/tiles/{TMS}` APIs are working

Added get_tile method for MVT-postgres for on-the-fly tile generation

029316e

- Supports properties - Supports both WebMercatorQuad, WorldCRS84Quad Co-authored-by: PRAJWAL S <praju18299@gmail.com> Co-authored-by: Tanvi Prasad <tanvi.prasad@datakaveri.org>

Added documentation for the MVTPostgresProvider class

a1fe9d5

ThorodanBrom force-pushed the add-mvt-postgres-provider branch from d9c05c6 to 97ba024 Compare May 2, 2025 19:05

Fix flake8 issues in base_mvt.py

6f22762

doublebyte1 reviewed May 8, 2025

View reviewed changes

doublebyte1 requested changes May 8, 2025

View reviewed changes

Fix MVT-postgres documentation - fix example config

e142aef

doublebyte1 approved these changes May 9, 2025

View reviewed changes

tomkralidis requested changes May 11, 2025

View reviewed changes

ThorodanBrom added 3 commits May 19, 2025 21:55

Changed name of MVT-postgres plugin to MVT-postgresql

9a320cc

- Updated all necessary files to change postgres -> postgresql related to the plugin - Updated documentation - Added authors to plugin file and organized imports

MVT-postgresql : Add check if the indices are not in limits of the TM…

a034d15

…S, return ProviderTileNotFound error - Otherwise error was thrown from PostgreSQL `ST_MakeTileEnvelope` function

Fix flake8 errors

9f596a1

ThorodanBrom force-pushed the add-mvt-postgres-provider branch from 19b9fbb to 9f596a1 Compare May 19, 2025 16:50

photonbit reviewed May 21, 2025

View reviewed changes

		LOGGER = logging.getLogger(__name__)


		class MVTPostgresProvider(BaseMVTProvider):

Uh oh!

Add provider for MVT tile generation from Postgres #1979

Are you sure you want to change the base?

Add provider for MVT tile generation from Postgres #1979

Uh oh!

Conversation

ThorodanBrom commented Mar 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Related Issue / discussion

Additional information

Dependency policy (RFC2)

Updates to public demo

Contributions and licensing

Uh oh!

doublebyte1 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

doublebyte1 commented Apr 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ThorodanBrom commented Apr 21, 2025

Uh oh!

doublebyte1 commented Apr 26, 2025

Uh oh!

tomkralidis commented Apr 27, 2025

Uh oh!

doublebyte1 May 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

doublebyte1 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ThorodanBrom commented May 8, 2025

Uh oh!

doublebyte1 commented May 9, 2025

Uh oh!

tomkralidis left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ThorodanBrom commented May 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

ThorodanBrom commented Mar 27, 2025 •

edited

Loading

doublebyte1 commented Apr 19, 2025 •

edited

Loading

doublebyte1 May 8, 2025 •

edited

Loading

doublebyte1 left a comment •

edited

Loading

ThorodanBrom commented May 19, 2025 •

edited

Loading