v2.5.2

Changed

• Bumped logging library to latest version which should fixed problem where containerized workload are not printing logs out in JSON format.

v2.5.1

This is a bug fix release containing a fix for inserting rows into a table for which no primary key constraint exist. For now, we still requires internally that your provide an id in your DatabaseChange of your row, a future update will lift that limitations.

v2.5.0

Highlights

This releases brings improvements to reported progress message while your Substreams executes which should greatly enhanced progression tracking

Note

Stay tuned, we are planning even more useful progression tracking now that we've updated progression data sent back to the client!

This releases also introduces a new mode to dump data in the database at high speed, useful for large amount of data insertion.

Substreams Progress Messages

Bumped substreams-sink v0.3.1 and substreams to v1.1.12 to support the new progress message format. Progression now relates to stages instead of modules. You can get stage information using the substreams info command starting from version v1.1.12.

Important

This client only support progress messages sent from a server using substreams version >=v1.1.12

Changed Prometheus Metrics

• substreams_sink_progress_message removed in favor of substreams_sink_progress_message_total_processed_blocks
• substreams_sink_progress_message_last_end_block removed in favor of substreams_sink_progress_message_last_block (per stage)

Added Prometheus Metrics

• Added substreams_sink_progress_message_last_contiguous_block (per stage)
• Added substreams_sink_progress_message_running_jobs(per stage)

New injection method

A new injection method has been added to this substreams-sink-postgres release. It's a 2 steps method that leverage COPY FROM SQL operations to inject at high speed a great quantity of data.

Note

This method will be useful if you insert a lot of data into the database. If the standard ingestion speed satisfy your needs, continue to use it, the new feature is an advanced use case.

See the High Throughput Injection section of the README.md file to check how to use it.

Added

• Added newer method of populating the database via CSV (thanks @gusinacio!).

  Newer commands:

  • generate-csv: Generates CSVs for each table
  • inject-csv: Injects generated CSV rows for <table>

v2.4.0

Changed

• gRPC InvalidArgument error(s) are not retried anymore like specifying and invalid start block or argument in your request.

• Breaking Flag shorthand -p for --plaintext has been re-assigned to Substreams params definition, to align with substreams run/gui on that aspect. There is no shorthand anymore for --plaintext.

  If you were using before -p, please convert to --plaintext.

  Note We expect that this is affecting very few users as --plaintext is usually used only on developers machine.

Added

• Added support for --params, -p (can be repeated multiple times) on the form -p <module>=<value>.

v2.3.4

Added

• Added logging of new Session received values (linear_handoff_block, max_parallel_workers and resolved_start_block).

• Added --header, -H (can be repeated multiple times) flag to pass extra headers to the server.

Changed

• Now reporting available columns when an unknown column is encountered.

v2.3.3

Fixed

• Batches written to the database now respects the insertion ordering has received from your Substreams. This fixes for example auto-increment to be as defined on the chain.

v2.3.2

• Fixed problem where string had unicode character and caused pq: invalid message format

v2.3.1

Fixed

• The substreams-sink-postgres setup command has been fixed to use the correct schema defined by the DSN.

• The cursors table suggestion when the table is not found has been updated to be in-sync with table used in substreams-sink-postgres setup.

Changed

• Now using Go Protobuf generate bindings from https://github.com/streamingfast/substreams-sink-database-changes.

v2.3.0

Added

• Added Composite keys support following the update in substreams-database-change

  The code was updated to use oneOf primary keys (pk and composite) to keep backward compatibility. Therefore, Substreams using older versions of DatabaseChange can still use newer versions of postgres-sink without problems. To use composite key, define your schema to use Postgres composite keys, update to latest version of substreams-database-changes and update your code to send a CompositePrimaryKey key object for the primary_key field of the TableChange message.

• Added escape to value in case the postgres data type is BYTES. We now escape the byte array.

Fixed

• Added back support for old Substreams Database Change Protobuf package id sf.substreams.database.v1.DatabaseChanges.

v2.2.1

Changed

• Reduced the amount of allocations and escaping performed which should increase ingestion speed, this will be more visible for Substreams where a lot of entities and columns are processed.

Fixed

• The schema is correctly respected now for the the cursors table.