What usecases do you have in mind for polars-config-meta

[paddymul]

What's the usecase you built this project for?

I'm building Buckaroo - a DataFrame table display library with summary stats. I also just wrote pl_series_hash that hashes series to a single u64 for use in caching. I'm planning to build out a caching system for buckaroo polars so that summary stats need to be computed only once and dataframes can be lazily displayed.

All of this seems similar to things you are building for polars-expr-hopper and polars-config-meta. Want to talk and see how we can collaborate?

[lmmx]

Oh hey, amazing! I’ve used Buckaroo not long ago, glad to be of assistance 😄

So this was a library to fill a need to store arbitrary metadata on a Polars DataFrame, there’s a GitHub issue on the Polars repo which I landed on while looking for it:

• Add DataFrame level metadata pola-rs/polars#5117

pl_series_hash sounds cool too!

If you run a code search on GitHub for polars_config_meta you’ll find where it’s been used (I think mainly by me!) — here

• octopolars
• polars-expr-hopper

The polars-expr-hopper was built to provide a simple (and deliberately limited) system for applying Polars exprs that met some condition. The purpose there was to provide a simple CLI that could pass in filters, in the case of octopolars the intention was to filter repos in someone’s GitHub account. The idea was a declarative sort of filter which would only be applied if it was applicable (i.e. if the dataframe schema had the column the filter would act on).

I had to review the code there to recall what the use for the metadata plugin was.

• Search results: All uses of the .config_meta namespace

The way it’s used in the polars expr hopper plugin is:

• When the DataFrame namespace initialises, it goes and gets the metadata from the dataframe that initialises it
  • This basically allows it to persist across dataframe expression calls. As I recall, if you call a regular dataframe method you can easily lose the metadata, so you have to use it in a deliberate way to retain it, or copy it across.
• The hopper has a data model of 3 types of columns that it stores as lists in the dataframe metadata, filters/selects/addcols, and these correspond of course to the Polars expr data model of filters/select/with_columns statements

As people pointed out in that Polars repo issue thread, this is not really a foolproof substitute for official Polars DataFrame metadata support, but in my view it’s still pretty good in absence of that 😃 You can also persist metadata to Parquet files themselves (and I’ve seen people do some impressive tricks with that like this).

There might be room for this to grow into the Rust extension side of things (especially if you’re trying to make it particularly performant), but so far it’s a pretty simple Python extension.

I’m currently working on a different Polars plugin, polars-genson for JSON handling, but that is a little different to this one (which is a simple Python only namespace plugin). That one is a Series handling plugin, with Rust code interfacing with the Python code.

I’ll check out your hashing plugin, and do feel free to open issues/ping me with an @ if I am not responding!

[lmmx]

I left a comment on your repo issue about possibly using rapidhash as a faster hashing library, but don’t quote me on that I’ve never used it, only heard about it this week 😄

I’d say polars-config-meta definitely sounds like the best way to store your cached columns for Polars yeah. For polars-expr-hopper, if I were you I would only use it if you were happy to let the expressions you are storing apply “as soon as they are ready” (i.e. as soon as the dataframe gets the columns that the expressions being stored refer to). It isn’t designed to apply them with logic besides that “when they’re ready” triggering, so if you’re after more than that it might not be sufficient for your needs. But it sounds like you want that when you say “lazy application”, so could be a good fit yeah 👍

[paddymul]

Hi. I'm revisiting this because I'm starting to write my cache bits. I really do hope that Polars implements metadata.

I just took a crack at trying to identify series that are re-used across dataframes.
so

    fc = FileCache()
    assert fc.check_series(df['a1']) == False
    assert fc.check_series(df['b2']) == False

    fc.add_df(df)
    assert fc.check_series(df['a1']) == True

    # buffer info for string series is unreliable commented out for now
    # look at polars-core/src/series/buffer.rs::get_buffers_from_string
    #assert fc.check_series(df['b2']) == True

    # I dont even see this reliably working
    #assert not fc._get_buffer_key(df['b2']) == fc._get_buffer_key(df['b2'])

    #this should show that the same physical memory is used by df2['a1'] as df['a1']
    df2 = df.select(pl.col('a1').alias('alias_a1'),
                    pl.col('b2').alias('alias_b2'))
    
    assert fc.check_series(df2['alias_a1']) == True
    #assert fc.check_series(df2['alias_b2']) == True

it works for numeric datatypes, but not strings, because strings don't consistently return the same buffer addresses.

Do you want to have a video call sometime to talk about what we both are building. I'm still not completely sure I understand how polars-config-meta fits in with what I'm trying to do, but it seems clsoe.

[paddymul]

So, would polars-config-meta work for this usecase:

With Buckaroo, I don't expect to have control of dataframes that are passed in. It feels like a big lift to my users to say "always load a dataframe with polars-config-meta".

Can I monkeypatch dataframes that are sent in?
so the following code

BuckarooWidget(polars_df)

# next cell

df2 = polars_df.with_columns(pl.col('profit').cumsum().alias('profit_cumsum'))

Because BuckarooWidget monkeypatched (not returned) a polars-config-meta onto the polars_df, df2 has the same metadata I also put in somewhere inside the BuckarooWidget call?

[lmmx]

I just took a crack at trying to identify series that are re-used across dataframes.

Very cool!

# buffer info for string series is unreliable commented out for now
# look at polars-core/src/series/buffer.rs::get_buffers_from_string

🧐

it works for numeric datatypes, but not strings, because strings don't consistently return the same buffer addresses.

🤔 💭

Do you want to have a video call sometime to talk about what we both are building. I'm still not completely sure I understand how polars-config-meta fits in with what I'm trying to do, but it seems close.

Sure, are you on Discord? I am in the Polars Discord server perhaps you are too? Username @permutans