remove s3 bucket polling

[MattShirley]

Added in preparation for front end changes per #1006

[ponyisi]

Looks good to me. I guess by default the index will be a B-tree and will be useful for the > operator?

[ponyisi]

Another question, for broader discussion - is it finally time for us to introduce a "capabilities" endpoint? In this case we could use it so the client could verify that this new endpoint is available and switch behavior appropriately. (Mostly I am starting to worry about different sites running different versions of the server software but all users picking up the latest versions of the clients.)

[MattShirley]

Looks good to me. I guess by default the index will be a B-tree and will be useful for the > operator?

Yeah, any index will be B-tree by default and I think it's good practice to over-index anything that might be used to filter values.

[MattShirley]

Another question, for broader discussion - is it finally time for us to introduce a "capabilities" endpoint? In this case we could use it so the client could verify that this new endpoint is available and switch behavior appropriately. (Mostly I am starting to worry about different sites running different versions of the server software but all users picking up the latest versions of the clients.)

Can these capabilities be reduced down to version numbers? If so, you could set the client version as a HTTP header, version the endpoints, and create a middleware to fail any request without a sufficient client version for the endpoint.

I spent some time hacking with Claude and think an interface like this should be possible:

class Endpoint(VersionedResource):
    pass

@Endpoint.require_client(3, 1, 1)
class EndpointV1_0_0(Endpoint):
    def get(self):
        return {"server_version": "v1.0.0"}

@Endpoint.require_client(3, 2, 0)
class EndpointV1_1_0(Endpoint):
    def get(self):
        return {"server_version": "v1.1.0"}
        
api.add_resource(Endpoint, '/my_versioned_endpoint')

[MattShirley]

@ponyisi @BenGalewsky I made major changes to this PR, please take a look. I noticed that files were available in the files table as soon as the request was made (and before results were available) while transformation_results streamed into our database as they became available. So now the endpoint checks transformation_results, which will be polled by the client. Please let me know if this change makes sense and whether we already have an endpoint that offers similar functionality.

[MattShirley]

@ponyisi @BenGalewsky added the feedback from Friday, including the cutoff/begin_at value to the endpoint.

[BenGalewsky]

Please contribute to the quality of our codebase with unit tests for this new endpoint

[MattShirley]

The feature flag has been added and tests updated/added. This is ready for release from my side.

[BenGalewsky]

Thank you! This is such a great contribution