HTTP 502 and 504 errors on some observations calls

[papakpmartin]

I'm seeing 502s and 504s when trying to hit a URL like https://api.weather.gov/stations/KFCM/observations?start=2021-09-28T15:59:00Z

I don't think it's related to the ?start=... as I'm seeing that if I remove the query arg.

Is this something that is known? Transient? Expected?

FWIW, other endpoints (like https://api.weather.gov/points/45,-93) seem fine.

[scadergit]

Not sure if related, but NCO is currently deploying a significant update to the platform. Please send outage issues to the NCO support email.

[scadergit]

There is an entire team dedicated to tier 1 support, so no worries. Unlike this forum, those are formally logged and reviewed by management, which means we can respond quicker as directed.

[jah1007]

We do apologize for the outage today. There were multiple failures of the operational system which were repaired at about 5:45 PM ET. As @scadergit eluded to there is a significant update coming tomorrow. While not the immediate cause it did prevent us from exercising our backup system today. Error codes should always be reported to operational support.

[papakpmartin]

No worries at all. I just thought I might be doing something wrong. Thanks for getting things fixed up! :)

[papakpmartin]

@jah1007 I'm noticing that the observations endpoint is returning a lot fewer observations now. I assume that the very large amount of observations previously visible (before yesterday) was probably a fluke, and I think there's not a commitment yet to how many observations we can expect long term, but I'm wondering if observations will now be limited to no earlier than when things recovered yesterday?

[jah1007]

So a few things here. The new release also came with a new database. Sparing the finer details, it will take time to fill it back to the seven days it should have. That said we encountered an issue when moving to the new release so the service was pointed at the old database. We hope to have an update in the coming days to move forward.