Regarding February 2022 Hubs Cloud Release

[robinkwilson]

Why was the February Hubs Cloud release so rough? And what did the Hubs Team learn as a result?

[robinkwilson]

Hello Hubs Cloud folks!

As the Hubs team, we’d like to address the roughness of the February 2022 Hubs Cloud release: what went wrong, what we learned, and what we’re doing to make releases smoother in the future.

Also, a resounding thank you to everyone in the Hubs community that gave us information, screenshots, and reproduction steps so our team could quickly hotfix issues within a few days after the release.

---

Here’s more information about why the February 2022 Hubs Cloud Release didn’t go smoothly this time:

1. The new packages required a template update, for the first time

The template updates included a new output parameter for the Admin Panel images and a required runtime update to the AWS Lambdas.

Luckily, these template updates are generally infrequent, but they are important and require a manual step that we can't automate inside AWS.

We admit that the one announcement in #announcements discord channel in October was insufficient to inform you that a template update was necessary leading to this release. Next time, we’ll be giving updates on Discord, Twitter, and any other channels we can find to ensure you all know there’s a required manual step.

2. This release accumulated more updates than usual

From the time we cut the release, we had accumulated about 5 months of changes. Because this was a large release, a lot of code was impacted. We have over 1000+ manual tests that we use for releases which we ran twice for this release. Once initially, then a second time after we fixed a few bugs QA surfaced.

3. We failed in persisting backwards compatibility for custom clients because of a new security feature that required custom clients to update

We’d like to drill something in here:

Custom clients should not break with Hubs Cloud releases.
We failed to uphold the backwards compatibility core tenant for this release.

Due to the size, complexity, and the very important security update in this release, testing backwards compatibility slipped through the cracks. We’re integrating more tests to directly catch this in the future. In the ideal scenario, we’d have had a smoother process to have you all update your custom clients.

We are still adamant that you merge the latest hubs-cloud branch code after each release because it can contain security fixes and will save you a code merge headache in the future.

Custom clients are one of our favorite features in Hubs Cloud since we get to see what you all create and innovate on. We’re sorry this release forced a merge abruptly and made this difficult.

4. We discovered we missed some manual test coverage: the custom client local development experience and Cloudflare setup

We’ve added new tests to double check that these features do not have issues with the new packages.

---

Ultimately, we’re sorry this was rough and are thankful for your patience and understanding. As a small team, we’re doing our best to make this process as smooth as possible and this time around we learned a lot. This release highlighted extremely important missing test coverage, places to improve our communication, and how thankful we are for such a supportive community.

As a teaser, we are currently working on a project to make having your own Hub easier. Please stay tuned in the #announcements channel in our Discord and follow us on Twitter.

We are so grateful for everyone’s patience, help, and overall positivity we see in our Hubs community.

Thank you all for your continued support,
The Hubs Team

[chris-metabi]

Thank you for all your hard work! I can't imagine what a PITA it must be.