Skip to content

[livelike.acs.puma] -- storing puma objects -- to/from class methods for pickle & parquet #40

@jGaboardi

Description

@jGaboardi

storing puma objects

to/from class methods

pickle & parquet

Teams idea dump


A idea I had yesterday that I need to put in a ticket is are to_*()/from_*() methods in {livelike,pronto_gatto}.acs.puma(). Where if the object is pickled ( puma.to_pickle(path) ) we can call acs.puma.from_pickle(path) and return the normal class instance.
The next logical step in thought is... how can we make a single parquet of a puma() class instance...

  • pandas.DataFrames can store a ton of different stuff
  • we can even store DFs/GDFs in cells of a DF
  • so... I think...
    • we can store puma class instances as parquet'ed DFs
    • singel column
    • indexed by attribute name
    • column values are attribute values
  • so the have another set of class methods to handle this
    • puma.to_parquet(path)
    • puma.from_parquet(path)

  • for parquet specifically
    • row- column based vs. column-based...?
  • structure
    • row 1 - name (str)
    • row 2 - weights (numpy.array)
    • row 3 - contraints (pandas.df)
    • row 4 - geoms (geopandas.gdf)
    • ....
  • We can (theoretically) store all attributes like this -- though maybe column-based instead of row based.
  • That way we would be able to store/ship around pre-built pumas in smaller format that pickle.

Big Qs

  1. Can this parquet solution be accomplished? I think yes, but not confirmed
  2. Can the parquet solution be a single column of rows vs. a single row of columns?

xref:

Metadata

Metadata

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions