▶️ Kaizen 18 - Pipely goes BAM - Feb. 28, 2025

[gerhard]

▶️ This is now available as changelog.com/friends/82 + 📝 episode transcript

---

This captures all things for the upcoming Kaizen episode.

Share your thoughts via comments below:

• Is there something in changelog.com that you, our dear listener, would like us to tackle?
• Is there something that you would like to help out with?
• Anything that we - @adamstac @jerodsanto - want to improve?

A rough timeline of what to expect (red line is today):

gantt
    title ▶️ Kaizen 18 timeline
    dateFormat  YYYY-MM-DD
    axisFormat %b %e, %Y

    section CONTRIBUTE
    💻 ALL CONTRIBUTIONS WELCOME :2024-12-13, 73d

    section RECORD

    🎙 Record :milestone, 2025-02-25
    ✂️ Edit :2025-02-25, 3d
    🚀 SHIP IT! :milestone, 2025-02-28

    section NEXT
    Kick-off Kaizen 19 :2025-02-28, 30d

Loading

---

What changed? Kaizen.17 ... NOW

• This is my Changelog episode 111
  • Yours?

---

• Make app resilient to Fly.io IAD region failures #543
  • The outage, as it happened - Feb. 16, 2025: https://community.fly.io/t/major-outage/23914
  • TL;DR what actually happened? https://community.fly.io/t/major-outage/23914/54
  • Final impact on changelog.com: https://community.fly.io/t/major-outage/23914/43
  • Our timeline: https://status.changelog.com/incident/513790
  • Lots of love in that thread for Fly.io logs & metrics

---

• Linkify chapters in Zulip new episode messages

---

• Upgrade Erlang, Elixir & Node.js to latest #541
  • DevOops
  • 👇 Want to talk about merge vs squash PRs?

---

• Why Why do you need a CDN if you have Fly.io? from Tim Uckun
• Minor updates to README, gitignore & Dockerfile pipely#4
  • First external contribution! 🙏 @mttjohnson
• Local Development pipely#5 👈 second issue
  • Pieces and notes for local development pipely#7 👍 and the resolution
• Resolving the Varnish TLS issue pipely#6
  • Someone new joins the 🪈 Pipely Party 🕺
  • Add support for TLS backends pipely#8
  • https://github.com/nabsul/tls-exterminator/

---

"Commit-driven company"

• Conference partnering + Fallthrough podcast
• 🦋 Bluesky
• Public podcasts in custom feeds

---

• Video podcasts!
  • Zulip discuss - @gerhard replied!
  • Add module that produces YT chapters from markers CSV
  • Add YouTube ID to top of published episode form
  • Add basic YT chapters support
  • 💙 Add YT episode redirects, e.g. changelog.fm/624/yt + 🧐 https://changelog.fm/624/yt
  • [admin] Improve YT description output
  • Add "Watch" buttons to episodes with YT videos
  • Embed and display YT video in page where applicable

---

• One more thing 😳

---

This is a follow-up to

• 🎧 Kaizen 17 - Three wise men? - Dec. 13, 2024 #525
• 🎧 Kaizen 16 - Just do it - Sep. 20, 2024 #520
• 🎧 Kaizen 15 - NOT a pipe dream- Jun. 28, 2024 #511

[gerhard]

Did you consider the build in test tool VTC ? from @ullgren

https://varnish-cache.org/docs/trunk/reference/vtc.html

[gerhard]

Why do you need a CDN if you have Fly.io? from Tim Uckun

Cross-posting here for visibility:

Top of my mind reasons mentioned in order of importance & complexity:

1. The changelog.com app is running close to the primary Postgres instance, not close to our users. Dealing with db latency + writers & readers is significantly more complex than putting a CDN in front of it.
2. The app is just one origin & we have multiple. The CDN acts as a proxy and makes it easy to migrate origins without any impact on users. For example, a few years back we switched from S3 to R2 for static assets. We could do it in a calm & predictable manner using a long blue-green strategy without our users changing anything.
3. Some app requests are expensive to serve. For example, the feeds are super heavy on the database since we generate all episodes that were ever recorded. The feeds grew so much, that they now get generated as background jobs that get pushed as static files to our object store. The CDN was updated to fetch them from there, rather than the app. From a practical perspective, if we only had the app to serve requests, or just the object store (regardless which one) this would involve user facing changing. A CDN abstracts away what happens "in the kitchen".
4. Some requests are 10x more efficient at the edge. Think all the redirects, throttling, etc. An object store like Tigris / R2 / S3 does not have that functionality, and running these in the app is inefficient.
5. A CDN should be able to serve 99% of the requests, and the app only 1%. This translates into a smaller resource footprint. It also simplifies the architecture.
6. When there are issues on Fly.io, or the db goes offline, we would prefer to continue serving stale requests rather than go offline. This doesn't always work as expected, since it's in the CDN provider's best interest to cache requests for as little time as possible. It means less RAM usage on their part. This is why we need to build our own CDN, so that we cache as much as possible, for as long as possible.

---