write_chunk and streaming

[kdesjard]

Is there a way to make write_chunk start sending data to the client right away? I'm trying to stream large amounts of data in a controller and it seems that write_chunk just adds to a body buffer and then sends all the data after the calls to finish and rendered. I end up with an out-of-memory error.

Thanks

[guest20]

Are you letting your chunk generation be governed by the "chunk written" callback, or are you just spamming calls to $c->write_chunk based on the read speed of your backend?

  # Make sure previous chunk has been written before continuing
  $c->write_chunk('H' => sub ($c) {
    $c->write_chunk('ell' => sub ($c) { $c->finish('o!') });
  });

mojo/lib/Mojolicious/Controller.pm

Line 892 in 75508c2

=head2 write_chunk

[kraih]

There is no way to block. You have to write one chunk at a time.

[kdesjard]

Ok, I'm doing that, but nothing is sent until the subroutine ends.

Eg. In a controller action:

sub myaction {
  my $self = shift;
  for(0..1000) {
    $self->write_chunk($buffer);
    # nothing is being sent
  }
  # nothing sent
  $self->finish;
  # nothing sent
}

[guest20]

Your for loop is blocking the whole-ass event loop, so nothing else in the event loop is going to happen until you fall of the end of sub myaction

[Grinnz]

See https://metacpan.org/pod/Mojolicious::Guides::Cookbook#Blocking-and-non-blocking-operations. The Mojolicious event loop and promises are designed on the principle of cooperative multitasking; except via something like the new async_await support using Future::AsyncAwait, you have to use callbacks or promises to queue up further code as the event loop will not regain control until each event handler completes.

[s1037989]

Rendering Guide for Streaming also seems relevant.

[Grinnz]

Apart from queueing the write_chunks as a callback and not a blocking loop, another option is to write your data to a temporary file asset and serve it with reply->file which will stream it in chunks automatically.

[kdesjard]

I think this is the best course of action. Thanks.