Quote ingest using apache stack: arrow / parquet

[goodboy]

In Follow up to #486, it'd sure be nice to be able to move away
from our current multiprocessing.shared_memory approach for
real-time quote/tick ingest and possibly leverage an apache
standard format such as arrow and parquet.

As part of improving the .parquet file based tsdb IO from #486
obviously it'd be ideal to support df appends instead of only full
overwrites 😂.

---

ToDo content from #486

pertaining to StorageClient.write_ohlcv() write on backfills and
rt ingest. rn the write is masked out mostly bc there's some
details to work out on when/how frequent the writes to parquet
files should happen, particularly whether to "append" to parquet
files: turns out there's options for appending (faster then
overwriting i guess?) to parquet, particularly using fastparquet,
see the below resources:

• for python we can likely use: https://fastparquet.readthedocs.io/en/latest/api.html#fastparquet.write

  • also note the times options with the int96 format which
    embeds nanoseconds B)
  • the custom_metadata: dict can only be used on overwrite 👀
    • can use the https://fastparquet.readthedocs.io/en/latest/api.html#fastparquet.update_file_custom_metadata
      to update metadata if needed?

• https://stackoverflow.com/questions/39234391/how-to-append-data-to-an-existing-parquet-file

• https://stackoverflow.com/questions/47191675/pandas-write-dataframe-to-parquet-format-with-append/74209756#74209756

• other langs and spark related:

  • https://issues.apache.org/jira/browse/PARQUET-1022
  • https://spark.apache.org/docs/1.4.0/api/java/org/apache/spark/sql/SaveMode.html
  • https://stackoverflow.com/questions/39234391/how-to-append-data-to-an-existing-parquet-file/42140475#42140475