-
Notifications
You must be signed in to change notification settings - Fork 0
Open
Open
Copy link
Labels
Description
storing puma
objects
to/from class methods
pickle & parquet
Teams idea dump
A idea I had yesterday that I need to put in a ticket is are
to_*()/from_*()
methods in{livelike,pronto_gatto}.acs.puma()
. Where if the object is pickled ( puma.to_pickle(path) ) we can call acs.puma.from_pickle(path) and return the normal class instance.
The next logical step in thought is... how can we make a singleparquet
of apuma()
class instance...
- pandas.DataFrames can store a ton of different stuff
- we can even store DFs/GDFs in cells of a DF
- so... I think...
- we can store puma class instances as parquet'ed DFs
- singel column
- indexed by attribute name
- column values are attribute values
- so the have another set of class methods to handle this
puma.to_parquet(path)
puma.from_parquet(path)
- for
parquet
specifically- row- column based vs. column-based...?
- structure
- row 1 - name (str)
- row 2 - weights (numpy.array)
- row 3 - contraints (pandas.df)
- row 4 - geoms (geopandas.gdf)
- ....
- We can (theoretically) store all attributes like this -- though maybe column-based instead of row based.
- That way we would be able to store/ship around pre-built pumas in smaller format that pickle.
Big Qs
- Can this parquet solution be accomplished? I think yes, but not confirmed
- Can the parquet solution be a single column of rows vs. a single row of columns?
xref:
- internal
livelike#-313
- internal
pronto_gatto#-120
- Migrate issues from GL -- GH #9