Proxy asset HREFs

[pjhartzell]

Related Issue(s):

• None

Proposed Changes:

Adds an asset proxy feature for generating pre-signed S3 URLs through the STAC API. When enabled, S3 asset hrefs on served STAC objects (via the API and Post-Ingest SNS Topic) are replaced with proxy endpoint URLs, with the original S3 URLs preserved via the Alternate Assets Extension. The proxy endpoints create and redirect to a pre-signed URL for the S3 asset being requested.

New Endpoints

• GET /collections/{collectionId}/items/{itemId}/assets/{assetKey} - Redirects to pre-signed S3 URL for item assets
• GET /collections/{collectionId}/assets/{assetKey} - Redirects to pre-signed S3 URL for collection assets

Configuration

• ASSET_PROXY_BUCKET_OPTION - Which buckets to proxy: NONE (default), ALL, ALL_BUCKETS_IN_ACCOUNT, or LIST
• ASSET_PROXY_BUCKET_LIST - Comma-separated string of bucket names (required when using LIST)
• ASSET_PROXY_URL_EXPIRY - URL expiry in seconds (default: 300)

Disabled by default.

PR Checklist:

• I have added my changes to the CHANGELOG or a CHANGELOG entry is not required.

[jkeifer]

This is great! But, also, a lot. 😁 I didn't make it into the tests or README. But I think I got through all the app code and that I left enough comments for the time being.

Happy to chat more if you want to discuss anything.

[jkeifer]

I'm going to go ahead and approve this. I think I am asking for one specific change around caching in the negative case (unless I misunderstand the code there). But I want to approve to unblock, and I would suggest pulling in @philvarner for a second pair of eyes and for awareness, assuming he has time for a review.

[matthewhanson]

Added some inline comments about the use of requester-pays in the docs and also:

1. The thumbnail link feature is now redundant (sort of). It creates a link with rel=thumbnail that fetches and returns the thumbnail by redirect via a pre-signed URL. However, STAC best practices suggest using a role of thumbnail to identify the thumbnail asset, so it is easily identifiable - so basically the thumbnail link is really a convenience function to identify the thumbnail asset. It doesn't seem too valuable, but since it's already implemented, and using the same getSignedUrl code I don't see much harm in keeping it. However, the documentation of the thumbnail link is poor - it only refers to thumbnail link generation with no details of what it's doing. This PR does not need to address this, but could be included in a future PR for refactoring the README which is getting overly long and complex. We should also decide if the thumbnail link is really worth keeping or should be deprecated since it's not in the spec nor best practices so it really is an implementation specific thing that isn't, and shouldn't be supported by clients.

2. I like the configuration of buckets, this seems like good practice to explicitly control which buckets stac-server shoud have access to. But if we look at the "80%" case the reality is the asset buckets will either be one bucket or one bucket for each collection. So this is really a feature to control proxy access for collections.

3. While the bucket configuration is useful what is perhaps more useful is the ability to control access based on asset keys. For instance I could proxy small assets - thumbnail, overview, metadata files, but not the actual data files. This feature would allow us to completely replace the thumbnail link feature. This would be useful on requester pays buckets because it allows small assets to be delivered for free while requesting the user pay for the larger data files. When Sinergise first made the sentinel-2 PDS requester pays they ended up setting up a separate API that proxied access to the tileInfo.json and thumbnail files, so that even public unauthenticated users could still get basic info about the scene.