trailing / or no trailing /

[chris-little]

This was Issue #584 in the Environmental-data-retrieval repo. We clearly want a consistent approach across the OGC APIs, so the issue is raised here.

The EDR API SWG are happy with the definition of the API URIs with no trailing / to indicate a resource, as the API is more about queries rather than a chain of sub-resources, and relative URIs are not used except for the Common parts.

For context and previous discussions see:
geopython/pygeoapi#97 (comment)

radiantearth/stac-browser#486

and a comment from the stac-broswer issue:
STAC Browser works with some OGC APIs, I tested it myself. It seems to depend on how the OGC APIs are implemented though, the spec seems to have some leeway there? Otherwise, it shouldn't work for some and not for others....

[jerstlouis]

@chris-little Our implementation considers the trailing / as equivalent to removing that trailing /.

Some servers in the past have required the presence of the trailing /, or have re-directed with a 3xx to the trailing / when not including it, which caused some problems with clients.

I think from how the various specs are defined that at minimum the servers should directly respond with a 200 when not including the trailing /, whether it redirects to no trailing / with 3xx or responds with 2xx whether you include it or not I think can be up to implementation.

[cportele]

@chris-little - I am not sure what the issue is.

All URIs of OGC API resources that I am aware of do not end with a "/" (there is one exception, see below).

A URI that ends with a "/" is a different URI. If an API accepts that, too, fine, but it does not have to. If it does and responds as if there was no trailing "/", the response still has to be based on the URI in the request. For example, where a relative URI is used these have to be based on the request URI (that is, the relative URIs differ between the with or without trailing "/" requests).

The exception mentioned above is the landing page, but that is a side-effect of the OpenAPI requirement that all paths must start with a "/", so we had to define the path as "/" although the preferred URI of the landing page should be without the trailing "/". This is the only resource where one can argue that the API has to accept the URI with the trailing "/".

[jerstlouis]

@cportele

where a relative URI is used these have to be based on the request URI (that is, the relative URIs differ between the with or without trailing "/" requests).

I think this is why we had decided to use relative URLs which are relative to the domain, but always starting from the root of the domain e.g., /ogcapi/collections (called "absolute path relative references" in RFC3986), to avoid those differences.

I am not sure what the issue is.

I think one issue is that over the years a lot of OGC API server implementations have linked to resources including the trailing /, and/or redirected to links including the trailing / when not included in the request, and/or return error when not including the trailing /, which would ideally not be the case and requirements/recommendations clarifying that in the various specs would be useful.

[m-mohr]

The issue is that in a generic client application (such as STAC Browser) that tries to construct URLs generically from self URLs without hardcoding a specific URL structure, you run into ambiguities. It would be simpler if the client hardcodes API endpoints such as /collections/:collectionid/items/:itemid, but this does not work when you have static catalogs (STAC or OGC API - Records). The URLs there can have any structure. So if you want a client that doesn't just supports APIs, you run into ambiguities.

For example, I get https://example.com/collections/123 as a self URL from a specific collection.
Then I'd like to construct the Item URL and try to add the item ID, so /items/567 as there's no self link in the Item returned in the item list.
Then I get for new URL('items/567', 'https://example.com/collections/123') which unfortunately results in https://example.com/collections/items/567.

But then another server returns https://example.com/collections/123/ as a self URL from a specific collection. I think both is allowed in OGC APIs.
Then I'd like to construct the Item URL and add again the item ID, so items/567.
Then I get for new URL('items/567', 'https://example.com/collections/123/') which correctly results in https://example.com/collections/123/items/567.

This is super unpredictable and makes creating URLs super difficult to implement with a lot of heuristics and special cases.

It's probably something like:

1. check whether you are in an API tree
   (unclear how to do that - e.g. /collections/ could also be part of a static catalog path)
   • then parse the base URL
     • ends with a slash? then get the collection ID and preprend it to /items/:id
     • ends not with a slash? then just use /items/:id
   • finally: resolve the path against the base URL
2. if not in an API path
   • get the base URL and resolve all relative links against it

But the first part is specific to an endpoint, in other endpoints you need to implement it differently again. It's a pain in the ass. Maybe I'm the only one stumbling across it yet as I'm the only one implementing a client for static and API usecases (i.e. for STAC and Records)?

Note: This issue only happens, if there's no self links, which is the case for example in https://reference.geoconnex.us/collections/hu10/items?f=json where the individual items don't have self links in the list. If the server would just return the links array for each item in the /collections/:collection_id/items endpoint, the issue would probably never face the light.

PS: Why is the links array for Features in OGC API - Features' OpenAPI schema optional while for a Feature it seems to be required that self links are provided? pygeoapi seems to omit links in the features that are returned in the .../items endpoint, which may or may not be valid?!? This leads to the whole mess in the first place, I think, because if self links would always be present, no URL construction/resolving would be needed in the first place.

PPS: An additional culprit could be that even the constructed URL by default may return non-JSON as the f=json query parameter is missing in the constructued URL (although I hope servers respect the Accept headers so that f=... is not needed.)

cc @RichFrazier @dblodgett-usgs Here are some more insights why it's so difficult (impossible?) to fix the STAC Browser issue.

[jerstlouis]

@m-mohr

Regarding:

Then I get for new URL('/items/567', 'https://example.com/collections/123') which unfortunately results in https://example.com/items/567.

I think RFC 3986 is quite clear about the rules in Section 4.2, and provides the Component Recomposition algorithm in Section 5.3 (and 5.4. Reference Resolution Examples). Specifically the two types of relative references: absolute path vs. relative-path:

A relative reference that begins with a single slash character is termed an absolute-path reference. A relative reference that does not begin with a slash character is termed a relative-path reference.

/items/567 is an absolute path relative reference, so it should never be used for such "partial path" (assuming these items are indeed inside a /collections/{collectionId} as in OGC API - Features).

From a /collections/{collectionId} path (which would normally not include the trailing /), I think the following relative path relative references need to be used:

{collectionId}/items/567 or ./{collectionId}/items/567

If the "self" URL ends with a /, then {collectionId} is omitted:

items/567 or ./items/567

hence the confusion with and without the trailing /, and why in our implementation we decide to stick to absolute path relative references, so that the same relative links can be used whether you access the resource with or without a trailing slash (the resolving should work exactly the same whether it's the HTML representation that a browser will resolve, or the JSON that a programmatic OGC API client will resolve).

[m-mohr]

Yeah, sorry, that leading / in the path segment was a late night error on my side. I meant to use it without the leading slash, updated above. Still, you see in the examples that it's not simple to construct these. The conditional leading {collectionId} is just a mess to handle as pointed out in my post above. Using the absolute path relative references only work reliably when you host API instances at the root, otherwise you'd need to add a path prefix to the root API location, which is yet another condition to cater for in the clients.

[dblodgett-usgs]

@cportele

This:

A URI that ends with a "/" is a different URI. If an API accepts that, too, fine, but it does not have to.

Is incompatible with the STAC best practice laid out here.

... To avoid issues it is recommended to consistently add a slash at the end of the URL if it doesn't point to a file.

I am not confident in this assessment because the nuances are a bit beyond my brain's ability to comprehend URL construction and API construct complexities, but I think we have two subtle and related issues here.

1. In OGC-API, item URIs are not explicitly discoverable through hypermedia. A client has to interpret the response document (geojson without a strict schema for encoding IDs that I'm aware of) to divine feature URIs and construct them with knowledge of a URL template that is not explicit in hypermedia.
2. Varied assumptions regarding the "next" link relation to discover and show items. This is the flip side of 1 in that, the basic expectation of OGC-API Features is that a client can page through the items as the base access mechanism to discover items. STAC expects that each item has explicit links and OGC-API doesn't.

I get the sense that a decision in OGC-API Features to relax requirements and add flexibility is ever so slightly out of sync with a decision in STAC-API / Stac-Spec. If we can identify the key workaround that will help generalized clients accommodate both assumptions, I think this issue is likely solvable. The question is "how to do it without introducing some horrible hack?"

[cportele]

@dblodgett-usgs

I don't think it is incompatible, it just states that there are differences between STAC URIs and OGC API URIs.

If an OGC Web API accepts requests to URIs that end in a "/" that is their decision. The OGC API standards do not require that. As I wrote in my first comment, no Collection, Features or Feature resource that implements OGC API Features has a trailing "/" in their URI. The requirements are clear. From an API design perspective, the current design follows good practices. Requiring that APIs must support two URIs for the same resource would smell like bad API design and would have made implementations more complex.

Static STAC catalogs are different based on the document that you have linked, which makes sense for folders and files simply published by a webserver.

To support clients that for whatever reason add trailing "/" to some URIs, APIs can be tolerant and accept a request to a URI with a trailing "/" and respond as if the requested URI was without the trailing "/" (or send a redirect), because some clients may use trailing "/". In any case, all links have to be correct, so absolute URIs or absolute paths are safe when sending the same response to different URIs (while relative URIs are not). This is also what we do in our implementation.

At the same time, client implementations that construct resource URIs following a pre-defined pattern should check, if the base URI includes a trailing "/" or not and adjust the URI construction accordingly. This is straightforward. Such a client should be able to browse both an OGC Web API and a static STAC catalog.

Regarding your item 1: We will address this in v1.1 of Features Part 1 by introducing a link template for feature URIs to simplify deriving the URI of a specific feature (by applying the feature id to the URI template). Requiring that each feature in a feature collection has individual link sets would be a bad idea, considering that there can be a very large number of features in a feature collection (including many link sets would significantly increase the payload) and since the derivation of the feature URI of each feature is really simple and straightforward. Also without that URI template in the response, clients should have no problems to construct the URI of a particular feature when they have the featureId and the base URI (of a Collection or Features resource).

Regarding your item 2: I do not understand what the issue is. If you follow the (opaque) "next" link, you get another feature collection with the next page, but that does not change how you construct links to individual features.

[dblodgett-usgs]

Thanks @cportele --

Pulling out a couple key points.

Static STAC catalogs are different based on the document that you have linked, which makes sense for folders and files simply published by a webserver.

...client implementations that construct resource URIs following a pre-defined pattern should check, if the base URI includes a trailing "/" or not and adjust the URI construction accordingly.

In one of my original questions about this I asked...

It seems like we should be able to use conformance / openapi to determine how to deal with trailing slashes in the case that we are dealing with OAF items vs STAC items. Is this not possible?

But then we went down an IETF relative vs absolute link relation rabbit hole that has evolved into this issue.

Given that STAC and OGC API are very similar but subtly different, I think the solution probably is helping people understand these subtle differences so they can traverse them without frustration. So critically, addressing @m-mohr's questions above #354 (comment) may get us where we need to be.

[m-mohr]

Just for visualization purposes, this is the excerpt of a code that is needed to (I think?) properly and reliably determine the URL for a Feature in a OGC API - Features that we need to request the feature from the endpoint where it's hosted, keeping in mind that there's static catalogs:

let selfLink = Utils.getLinkWithRel(item.links, 'self');
let url;
if (selfLink?.href) {
  url = Utils.toAbsolute(selfLink.href, baseUrl, false);
}
else if (typeof item.id !== 'undefined') {
  let apiCollectionsLink = cx.getters.root?.getApiCollectionsLink()?.href;
  if (apiCollectionsLink) {
	apiCollectionsLink = new URI(apiCollectionsLink);
  }
  if (baseUrl && baseUrl.path().endsWith('/')) {
	url = Utils.toAbsolute(`items/${item.id}`, baseUrl, false);
  }
  else if (baseUrl) {
	url = Utils.toAbsolute(`${collectionId}/items/${item.id}`, baseUrl, false);
  }
  else if (apiCollectionsLink?.path().endsWith('/')) {
	url = Utils.toAbsolute(`${collectionId}/items/${item.id}`, apiCollectionsLink, false);
  }
  else if (apiCollectionsLink) {
	url = Utils.toAbsolute(`collections/${collectionId}/items/${item.id}`, apiCollectionsLink, false);
  }
  else if (cx.state.catalogUrl) {
	url = Utils.toAbsolute(`collections/${collectionId}/items/${item.id}`, cx.state.catalogUrl, false);
  }
  else {
	return null;
  }
}
else {
  return null;
}
return url.toString();

This is in STAC Browser and I expect similar clients to require similar code. And that's just for Items, we haven't talked Collections yet.

The whole checking whether there's a slash at the end is a pain. I think there can't be any opinion that this is a good as is, right?!?

[webb-ben]

• In OGC-API, item URIs are not explicitly discoverable through hypermedia. A client has to interpret the response document (geojson without a strict schema for encoding IDs that I'm aware of) to divine feature URIs and construct them with knowledge of a URL template that is not explicit in hypermedia.

While not normative, the default JSON-LD representation of /items endpoints from pygeoapi serve the FeatureCollection as a list of Feature URIs. See: https://reference.geoconnex.us/collections/hu02/items?f=jsonld or https://demo.pygeoapi.io/master/collections/obs/items?f=jsonld.

[dblodgett-usgs]

+1 to that -- the broader point is that the hypermedia driven aspect of OGC-API normative content is incomplete on the point of individual item URIs where STAC more or less expects a complete set of hypermedia for individual items. However, STAC is much less specific about its rules for specification (it doesn't conform to the modular spec) and has a lot of "best practice" type statements for specific behavior that is not strictly normative but is nevertheless the "norm".

[dblodgett-usgs]

@cportele -- this comment of yours has stuck with me...

I don't think it is incompatible, it just states that there are differences between STAC URIs and OGC API URIs.

You realize that "differences between" and "incompatible" are two ways of saying the same thing, right?

STAC is currently up for adoption as a community standard and is advertised as:

To the greatest extent possible, STAC uses and extends existing specifications. The most important object in STAC is an Item, which is simply a GeoJSON Feature with a well-defined set of additional attributes ("foreign members"). The STAC API extends the OGC API - Features - Part 1: Core with additional web service endpoints and object attributes.

(from https://github.com/radiantearth/stac-spec?tab=readme-ov-file#about)

It's critically important that we can clearly describe how or if these specifications are compatible for the sake of community members who are frustrated that things seem to work together to a degree but fail in important ways -- such as when constructing links to resources.

I'm not sure what the solution is here, but it feels like, at a minimum, some informative content describing how to bridge between the static file pattern (where a trailing slash is a folder and everything else is a file) and the API pattern (where a trailing slash is undefined and no trailing slash is an API endpoint) would be helpful to the community.

[cportele]

@dblodgett-usgs

You realize that "differences between" and "incompatible" are two ways of saying the same thing, right?

No, "compatible" is not the same as "equal". If a client can support both STAC URIs and OGC API URIs at the same time, I consider them a "compatible" . "Incompatible" would imply to me that this is not possible.

Also note that STAC resources that are accessible via a STAC API will be accessible via the OGC API URIs without trailing "/" (at least this is my understanding, which is consistent with the sentence you have quoted).

I agree, it seems it could be helpful, if someone volunteers to document guidance how a client can access static STAC catalogs, dynamic STAC catalogs and OGC Web APIs at the same time.