HTTP vals are far too slow

[bitsoflogic]

I'm not sure how you can solve this at the moment given this.

GET / takes ~210ms+ (sometimes 500-900ms)
ping / takes ~10ms

The GET / http handler:

export default async function(req: Request): Promise<Response> {
  return new Response("");
}

Having 200ms of overhead for every request is just too much! That's before I do any processing of my own. It's a poor user experience.

Example 1: https://www.val.town/v/greatcode/htmx
Example 2: https://www.val.town/v/neverstew/htmxExample (Requests here are ~350ms with very little extra added to my above example)

P.S. Also, "cold starts" appear to take 500ms+. I had one cold start take nearly 1s with the above code.

[postpostscript]

This is worse still with large response sizes. Not the biggest deal for hobby projects but it's a noticeable slowdown, especially if a val makes a request to another val

[stevekrouse]

Quick update - @maxmcd is actively working making larger responses faster by skipping an encoding/decoding step. More soon!

[robinsloan]

It occurs to me that this might be something to consider in the context of the Pro tier; maybe there's a nice congruence between user desire/demand and infrastructural overhead. TurboVals!

[saolsen]

I have some pretty big vals and performance is starting to become an issue. I think for me most of the overhead is that everything is created from scratch every time. I'm timing my handlers and for this specific val it responds in about 200ms, but the full request takes 1-2 seconds.

If I also time the imports (By turning them all into const {blah} = await import("lib")) I get about 700ms which accounts for some of the 1-2 seconds but not all of it.

If I understand correctly, vals are loaded and instantiated for each request, so each request is effectively a "cold start", doing all the val.town side loading as well as imports / initialization. I think if vals could be cached in memory (or something) and respond to multiple http requests it would speed it up a lot.

Obviously, that's way more complicated but I'm really just here to +1 this discussion and am excited to see what improvements you all make.

[stevekrouse]

Can you provide links to specific vals? Would be helpful for benchmarking!

[saolsen]

Def. I think a val like this one should be fast.

https://www.val.town/v/saolsen/traces

You can see the val responds in 116ms https://saolsen-traces.web.val.run/v/saolsen/traces/t/b4f24f9b0b39be7dec4edca7688eff1f/s/f85f1d3a81017ac4, but the network request takes 125 seconds

[stevekrouse]

We (mostly @maxmcd) are working on this! We are currently working on figuring out where things are slow. We are hopeful that there will be some quick wins🤞. We'll report back soon!

[tomByrer]

status please?

[stevekrouse]

We got a huge speedup! Results below: #113 (reply in thread)

[maxmcd]

Hello! I'm working on performance improvements of vals and wanted to provide a quick update on the work so far.

The use-case I'm focusing on at the moment is a dead-simple http val:

export default async function(req: Request): Promise<Response> {
  return Response.json({ ok: true });
}

If I publish this val and make a request to https://maxm-examplehttp.web.val.run/ this request will complete in, at best, around ~190ms. Here's a breakdown of where that time is spent:

• 190ms Total Request time
  • 90ms Network overhead as request travels to our host
  • 100ms Request processing on our host
    • 20ms Database calls ahead of val execution
    • 78ms Running Deno
      • 72ms Fetching the val source
      • 6ms Deno handles the request
    • 2ms Serializing and returning the response

These numbers are rough estimates because they can vary quite a bit, but I think this is representative of where time is typically spent.

A few takeaways from this breakdown:

1. We have a bunch of unexplained network overhead on requests and when fetching the val source. This shouldn't be present and we're working on figuring out what's causing it so that we can remove it.
2. There's a database querying cost ahead of each request. 20ms is a bit much. We should be able to make this much smaller.
3. Serializing and communication overhead of talking to Deno is fast in this simple case, but can be much larger for larger val responses. We'll need to focus on that in the future.

Notably, when we fetch the val source code (the 72ms step above) we make a request to "https://esm.town/v/maxm/examplehttp" which makes a full round-trip out to the public internet. We can fix this directly and avoid the networking round-trip entirely, but for the moment we're working on figuring out why that amount of network overhead is present at all. This overhead is present in all esm.town requests so vals with lots of imports are also slow at the moment and should see large speedups when we've shipped improvements here.

Happy to answer any questions or take suggestions on how to proceed with these improvements 🙂

[maxmcd]

Quick update: We shipped the improvement I mentioned above and requests to esm.town are not longer making a long trip out out of our server for every request. This speedup is particularly noticeable in more complicated vals with lots of imports where we're seeing requests complete 2x faster.

I benchmarked the page @saolsen linked to, and we're seeing a solid speedup:
Before:

$ h2load -n 6 -c 3 https://saolsen-traces.web.val.run/v/saolsen/traces/t/b4f24f9b0b39be7dec4edca7688eff1f/s/f85f1d3a81017ac4
                     min         max         mean         sd        +/- sd
time for request:    1.28s       6.83s       2.74s       2.29s    83.33%

After:

$ h2load -n 6 -c 3 https://saolsen-traces.web.val.run/v/saolsen/traces/t/b4f24f9b0b39be7dec4edca7688eff1f/s/f85f1d3a81017ac4
                    min         max         mean         sd        +/- sd
time for request:   685.54ms    856.05ms    774.66ms     62.17ms    66.67%

There's much more to do to make things faster, but this should certainly make things a little snappier.

[maxmcd]

Hello!

There is finally a path to very fast HTTP vals. Wanted to give a quick update about how we're going to do that.

1. We're going to re-use Deno processes for follow-on requests. This will mean that Val Town will work more like other FAAS providers where the first request starts the process and subsequent requests re-use the existing process. In local testing we've seen requests take as little as 5ms with this approach and we can likely get that even lower.
2. We're going to start locking dependencies per-version. This change gets us security, stability, and performance wins. When you save a val we will generate a deno.lock file and cache dependencies. On future runs we'll start the Deno process using the generated lockfile. This will save us from making many network round trips and allow Deno to resolve dependencies much more efficiently.

Today, a val spends most of its execution time starting up and loading dependencies, so these two features should make things very speedy.

This doesn't solve everything, certain patterns will still be slow. Cold-starts will still be a bottleneck, we still have some network latency added by our current hosting provider, and requests will be slow if you are far away from Ohio. We'll keep improving things in these areas and push for things to to be faster and faster, but we're optimistic that these two features will lead to a large speedup for common HTTP val request patterns.

[bitsoflogic]

This is really great news! Well done, and thank you for keeping us all in the loop!

[trentguillory]

This is very exciting, Max. Keep us updated - this would be a game changer.

[stevekrouse]

HTTP vals are very fast! ~50ms to get a reply from a basic val here in NYC

We can (and will) make them faster but they are no longer "far too slow", so closing this ticket

[maxmcd]

To summarize the improvements here:

• We've finished the work I referenced in previous messages. We re-use the deno process and we have lockfiles.
• Our network is much much faster, routing to our datacenter and val-to-val calls are much faster.
• Thanks to lockfiles and some other improvements we are much better at caching dependencies.

There is much more performance to unlock. We're working on more re-architecting and I hope that in the near future a warm Deno process will respond in <5ms.

I think the issues highlighted in this ticket are now solved. If you have more specific performance needs, or you're running into other slowdowns on our platform please open a new discussion!