How can I use Marvin to return a large dataframe or other data

[webcoderz]

Hi , trying to figure out how I can utilize Marvin to send back either chunked or batched responses , the use case I’m thinking of is having a tool or something that can do a data transform which results in a large dataframe (or similar)and I want to return the large dataframe that may not entirely fit into the context of a single model call .. any clue how I might approach this?

[zzstoatzz]

hi @webcoderz - i added an example that might be helpful in #1056

specifically in the sense that you ask for whatever result_type so that you can chunk your df however you want, run marvin calls concurrently and stitch the results together if needed. its possible we'd support a more native df thing behind some marvin[pandas] extra

[webcoderz]

Amazing yea I was thinking like a scatter and gather approach would work best also