Fetch Multiple TimeSeries via TimeSeries Group

[krowvin]

Proposal

Allow retrieving timeseries data from a provided timeseries group.

Reasoning

cwms-python, java, and other languages can take advantage of multiple threads. Browsers appear to cap AJAX requests to 6 threads (for Chrome). In my limited research the way around this is through Domain Sharding which goes on to say if you use HTTP/2

HTTP2 supports unlimited concurrent requests making domain sharding an obsolete requirement when HTTP/2 is enabled.

More here on HTTP/2 - also of interest maybe the ability to

Server Push, which allowed a server to send resources to a client in anticipation that the client would need them very soon.

We appear to be using 1.1 now:

curl -s -o /dev/null -w "%{http_version}\n" "https://cwms-data.usace.army.mil/cwms-data/timeseries?name=KEYS.Elev.Inst.1Hour.0.Ccp-Rev&office=SWT"
1.1

Considerations

• Allow users to specify a time window
• Write large sets of data to the blob endpoint on a schedule either in static report or JSON format and query that
• /timeseries/recent does seem to allow for group retrieval or multiple timeseries, but this is only for recent data
• Using HTTP/2 would potentially require configuring jetty, and out of the box http/2 is not enforced
• Clients, proxies, and other midpoints may not support 2 yet. Could probably do both and let client negotiate which to use? I.e. browsers supporting 2.0 for parallel +6.

[rma-rripken]

In cwms-mobile I had a method to retrieve multiple timeseries via timeseries group. It makes sense to me. Time series retrieval can be slow so you still want to be smart with the time-windows. Clients can split the timewindow so that multiple clients make many of the same requests and results will likely be in server cache.

[krowvin]

Clients can split the timewindow so that multiple clients make many of the same requests and results will likely be in server cache.

Indeed, and we still have additional caching to talk about.

More conversation on benefits of not relying on cache

There are workarounds to edge performance but you still end up in a situation where each request at some point has a first query.

Then take this and multiply it across all districts with each report (Dashboard/plot/table/etc) possibly having 30+ timeseries ID.

See how they trigger 6 at a time in parallel and will still stall at 200-300ms each - this is the 5th load of the page with cache disabled on the client-side

This is an example of a report I worked with SWL on that was meant to replace a static (text based?) report (I believe). It uses timeseries groups/location groups to decide with TS/locations to use.

[MikeNeilson]

There's also the idea of a "task list" were you schedule the retrieval job in the background and return a unique identifier back to the user to poll for the data when ready. That's really the "right" way to handle bulk data and one of the async methods we've discussed in the past.

As for HTTP/2, we should figure out how to support that. @willbreitkreutz I vaguely remember the CWBI WAF having issues supporting HTTP/2, but I might be mixing that up with the G6 DMZ WAF. Do you know if the CWBI WAF supports HTTP/2? if so we may just have forgotten to make sure the required tomcat config element is present in our container.

[rma-rripken]

If this was accomplished as a new end-point - what would the payload look like? You get back the timeseries group ( listing the assigned timeseries) and then a map of ts-id to TimeSeries ? Would it need to support paging? I could imagine the paging getting a little hairy.

Really would be great (for sanity) if there was just a switch that could be flipped on the server config to get us http/2

[MikeNeilson]

For tomcat, it's a simple addition of an element to allow upgrade, but every proxy or middleman in the path has to support HTTP/2 or the upgrade will fail.

CDA's docker image has it enabled:

cwms-data-api/cwms-data-api/src/docker/server.xml

Line 51 in 976fad3

<UpgradeProtocol className="org.apache.coyote.http2.Http2Protocol" />

I think we also have that on the district systems, so user direct to that Tomcat instance could have http/2.

I don't remember it working on the original DMZ system, and we're currently waiting on @willbreitkreutz response.

However, if the task list/queue is used, pagination isn't required. The entire request would've been processed and can be streamed to the user. the 10 second timeout is idle time, not total time, said large request would be a continuous download and not subject to that timeout.

[willbreitkreutz]

@MikeNeilson I also vaguely remember issues with HTTP/2, but will need to confirm with Blake, I'm seeing some other modules trying to use it so I think it may be supported now.

[MikeNeilson]

@willbreitkreutz while were at it... websockets would also be useful.

[willbreitkreutz]

@MikeNeilson yup, totally agree