🎧 Kaizen 12 - Slightly more instant - Oct. 13, 2023

[gerhard]

🎧 This is now available as changelog.com/friends/17

See all improvements in this 🐙 GitHub repository: Kaizen 11 ... 12

---

This contains all things related to the next Kaizen episode.

Share your thoughts via comments below:

• Is there something in changelog.com that you, our dear listener, would like us to tackle?
• Is there something that you would like to help out with?
• Anything that we - @adamstac @jerodsanto - want to improve?

A rough timeline of what to expect (red line is today):

gantt
    title 🎧 Kaizen 12 timeline
    dateFormat  YYYY-MM-DD
    axisFormat %b %e, %Y

    section DISCUSS
    ✍️ Add your comments below :2023-08-11, 61d

    section CONTRIBUTE
    💻 ALL CONTRIBUTIONS WELCOME 👕 :2023-08-11, 61d

    section RECORD


    🎙 Record :milestone, 2023-10-11
    ✂️ Edit :2023-10-11, 3d
    🚀 SHIP IT! :milestone, 2023-10-14

    section NEXT
    Kick-off Kaizen 13 :2023-10-14, 14d

Loading

---

Proposed topics

What we didn't do: multi-node clustering & caching

• Did procrastinating pay off?

Serving all our feeds from R2, via Fastly

• How did that go Jerod?
• Pairing sessions with James A. Rosen - @jamesarosen
  • We were holding AWS S3 & Fastly integration wrong - config now removed from Fastly)
• Cached /feed tail latency improved by 4x, and cache hit ratio improved by 3x
• See the GitHub discussion for more details
  • Fastly debugging tips
  • Analyzing HTTP response headers
  • Honeycomb.io before & after screenshots
  • James's CacheHitRatio formula of type math Markdown
• SLO 96% of all feeds serve within 1s - last 60 days - notice the gain since Aug. 31, 2023

Oban Web - PR #472

• Well I finally got Fastly websockets passthrough configured correctly and now Oban Web is working like expected
  • Your free trial of WebSockets ends on October 29, 2023
• Big shout-out to Parker Selbert - @sorentwo
• Check the debugging chat in our Slack #dev channel - don't archive it!
• Small shout-out to Lars Wikman - @lawik
• See PR Install and configure Oban Web #472 for all the details, including how to integrate it in a Dagger CI/CD pipeline

Migrating nightly.changelog.com from DigitalOcean to Fly.io

• 🤖 Daggerize & deploy to Fly.io ✈️ nightly#42
  • Containerize & Deploy, a.k.a. daggerize
• Cloudflare Pages experiment
  • wrangler pages project list
  • https://nightly.changelog.place
• Cloudflare R2 experiment - index.html
• Resolves nightly.changelog.com doesn't support IPv6 #420

Honeycomb.io

• SLOs vs Triggers (see the GitHub discussion for more details)
• Elevated 5xx HTTP responses trigger - 15mins
• Elevated 404 HTTP responses trigger - 60mins
• SLO 99% of all responses are 2xx or 3xx - last 60 days - handle some of those 404s
  • Reduce 404 responses - 1.46% of all traffic

Fastly

• Enable HTTP/3 QUIC
  • Noticed the absence of HTTP/3 responses in Fastly's beta Edge observer
• Use Brotli compression when available
  • 10-15% reduction in traffic over gzip - How can we measure that?
• Now redirecting /rss at the edge - 400ms vs 1ms

KubeCon NA 2023

• Meeting for the first time in person - 7 years in the making!
• Who do we want to record with?
  • Frederic from Polar Signals - recently launched PolarSignals Cloud
    • Ship It! #57 was the last time Gerhard recorded with him
    • Find previous conversations with Frederic on Changelog - starting with KubeCon 2019!
    • Our Fly.io experiment with PolarSignals Cloud: https://www.polarsignals.com/docs/setup-collection-kubernetes/
  • Erik, a BuildKit maintainer, that has been doing some really cool things with distributed caching
    • S3 R2 B2 on steroids
  • Solomon!
• Dagger @ KubeCon!
  • Come say hi to Gerhard at booth N37

---

What changed so far? Kaizen.11 ... NOW

• Limit app traces sent to Honeycomb.io #482
• Install and configure Oban Web #472
• Add new caching variant for multi-node clustering #451
• 🤖 Daggerize & deploy to Fly.io ✈️ nightly#42

---

What else is coming up?

• Parca on Fly.io -> PolarSignals Cloud - https://www.polarsignals.com/docs/setup-collection-kubernetes/
• Reduce 404s
• Experiment with middleware.io
• Elixir / Phoenix releases
• 1Password Service Accounts
• IPv4-only S3 endpoint is used for post pictures #419

---

This is a follow-up to

• 🎧 Kaizen 11 - S3 R2 B2 D2 - Aug. 11, 2023 #469
• 🎧 Kaizen 10 - The best pipeline ever™️ - Jun. 2, 2023 #452
• 🎧 Kaizen 9 - Embracing change 🌟 - Mar. 2, 2023 #440
• 🎧 Kaizen 8 - 24 improvements & a lot more - Nov. 23, 2022 #433

[gerhard]

This is a placeholder thread for talking fastly.com integration async with @jamesarosen: https://changelog.slack.com/archives/C03SA8VE2/p1692485028316819

[gerhard]

Changelog & R2

As a first step, we ensured that we are returning the correct header to Fastly so that static assets coming from R2 will get cached correctly. This is the end-result:

The cache-control header that we set in Cloudflare for all changelog.place responses is saying:

• max-age=3600 - clients should cache for 1h
• s-maxage=604800 - a proxy cache (e.g. Fastly) should cache for 7d
• stale-while-revalidate=604800 - cache can serve a stale response for another 7d while it fetches the object from the origin, typically in the background
• stale-if-error=604800 - cache can serve a stale response for 7d if the origin returns an error (usually a 5xx response)

The other 2 headers in the above screenshot are:

• cf-cache-status: HIT - this response was served from the Cloudflare cache
• age: 36 - it was stored in the Cloudflare cache for 36s ago at the time the response was served

FTR:

Fastly

Now let's see what Fastly is returning for the same url:

We see that the original cache-control headers set in Cloudflare are preserved.

We also see that the request was not cached in the Cloudflare POP that serviced the Fastly request.

The headers highlighted in cyan are most interesting:

• x-served-by: cache-iad-kjyo7100132-IAD, cache-lon420136-LON - request was served by the Fastly LON edge POP (closest to my location), which requested it from the Fastly IAD POP (our cache shield)
• x-cache: MISS, HIT - request was served from the edge cache (the second HIT), and it was not in the shield cache at the time

Note
This last part is really important. While this might look like a miss, meaning that it was not served from the Fastly cache, only the initial request to the cache shield - IAD - was a miss. If the edge cache has it - LON -, then the cache shield loaded it too when the edge requested it.

Adding Fastly-Debug: 1 header shows us more:

This Fastly.com doc page helps with understanding the symbols.

mp3 responses from cdn.changelog since R2

WIP

AWS S3

WIP

---

Links:

• https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control
• https://docs.fastly.com/en/guides/checking-cache
• https://developers.cloudflare.com/cache/how-to/cache-rules/

[jamesarosen]

x-cache: MISS, HIT - request was served from the edge cache (the second HIT)

Correct.

and it was not in the shield cache at the time

This is either correct or incorrect depending on the reading of the sentence. It was not in Shield cache at the time Edge first requested it from Shield and then placed it into its cache. It may or may not be in Shield at the point when Edge is serving a HIT, but Edge doesn't know because the HIT means it doesn't bother connecting to Shield.

[jamesarosen]

Cache-Hit Ratio

If you want to know how much work Fastly (Edge+Shield combined) is offloading from R2, you want

$$CHR = \frac{(HIT,HIT) + (HIT,MISS) + (MISS,HIT)}{(HIT,HIT) + (HIT,MISS) + (MISS,HIT) + (MISS,MISS)}$$

The Fastly-Honeycomb integration treats the user-to-edge and edge-to-shield requests as separate, so to compute the ratio in Honeycomb, you need to deduplicate those requests.

# this counts all requests served from Edge
# aka (HIT,HIT) + (MISS,HIT)
count (where cache=HIT and origin like "Shield")

# this counts all requests served from Shield
# aka (HIT,MISS)
count (where cache=HIT and origin like "Cloudflare")

# this counts all requests served from Origin
# aka (MISS,MISS)
count (where cache=MISS and origin like "Cloudflare")

Thus

$$CHR = \frac{\text{count (where cache=HIT)}}{\text{count (where cache=HIT or origin like "Cloudflare")}}$$

or, equivalently,

$$CHR = 1 - \frac{\text{count (where cache=MISS and origin like "Cloudflare")}}{\text{count (where not (cache=MISS and origin like "Shield"))}}$$

[gerhard]

Since we implemented that cache-control last week in R2, this is how our IAD shield GET /master/feed request caching changed: https://ui.honeycomb.io/changelog/datasets/fastly/result/57f88pV53K1

Since then, we had 435 shield cache hits & 18 misses, meaning 96% cache ratio.

Before, we had 100 shield cache misses & 81 hits, or 45% cache ratio.

If we look at all GET /*/feed requests, cache misses dropped by 7x - 9k vs 69k: https://ui.honeycomb.io/changelog/datasets/fastly/result/c93uNo49nna

As for the response time improvement, P99 for feed cache hits went from 2.8s to 1s: https://ui.honeycomb.io/changelog/datasets/fastly/result/meSXvSDkTcN

Since we started serving pre-computed feeds from R2, p99 responses for feed misses got worse, from 6.7s to 7.2s: https://ui.honeycomb.io/changelog/datasets/fastly/result/meSXvSDkTcN

It's not bad news though! Tail latency improved though - 121s vs 528s: https://ui.honeycomb.io/changelog/datasets/fastly/result/DxFktQBbPw9

---

Question for @jerodsanto: should GET /feed be served by the app, or R2? https://ui.honeycomb.io/changelog/datasets/fastly/result/zbprxo8Re4B

Question for @jamesarosen: anything that I wrote above which you would like us to dig into?

[gerhard]

https://twitter.com/gerhardlazu/status/1704438524705915352