v0.29.0

$ docker pull graphprotocol/graph-node:v0.29.0

Upgrade notes

• This release includes a determinism fix that affect a very small number of subgraphs on the network (we counted 2): if a subgraph manifest had one data source with no contract address, listening to the same events or calls of another data source that has a specified address, then the handlers for those would be called twice. After the fix, this will happen no more, and the handler will be called just once like it should.

  Affected subgraph deployments:

  • Qmccst5mbV5a6vT6VvJMLPKMAA1VRgT6NGbxkLL8eDRsE7
  • Qmd9nZKCH8UZU1pBzk7G8ECJr3jX3a2vAf3vowuTwFvrQg

  Here's an example manifest, taking a look at the data sources of name ERC721 and CryptoKitties, both listen to the Transfer(...) event. Considering a block where there's only one occurrence of this event, graph-node would duplicate it and call handleTransfer twice. Now this is fixed and it will be called only once per event/call that happened on chain.

  In the case you're indexing one of the impacted subgraphs, you should first upgrade the graph-node version, then rewind the affected subgraphs to the smallest startBlock of their subgraph manifest. To achieve that, you can use the following graphman rewind CLI command invocations:

  graphman rewind 0xd75c2a2412396b6b2b387323d84b9c19313dbba35ccb1493d94976fedb0b3a23 5774644 Qmccst5mbV5a6vT6VvJMLPKMAA1VRgT6NGbxkLL8eDRsE7
  graphman rewind 0xf23c45dd2e1ed77fce1919cdcc9df2de51e821763e9691f8adbb79a55ddb28a4 3914495 Qmd9nZKCH8UZU1pBzk7G8ECJr3jX3a2vAf3vowuTwFvrQg
  

  See #4055 for more information.

• This release fixes another determinism bug that affects a handful of subgraphs. The bug affects all subgraphs which have an apiVersion older than 0.0.5 using call handlers. While call handlers prior to 0.0.5 should be triggered by both failed and successful transactions, in some cases failed transactions would not trigger the handlers. This resulted in nondeterministic behavior. With this version of graph-node, call handlers with an apiVersion older than 0.0.5 will always be triggered by both successful and failed transactions. Behavior for apiVersion 0.0.5 onward is not affected.

  The affected subgraphs are:

  • QmNY7gDNXHECV8SXoEY7hbfg4BX1aDMxTBDiFuG4huaSGA
  • QmYzsCjrVwwXtdsNm3PZVNziLGmb9o513GUzkq5wwhgXDT
  • QmccAwofKfT9t4XKieDqwZre1UUZxuHw5ynB35BHwHAJDT
  • QmYUcrn9S1cuSZQGomLRyn8GbNHmX8viqxMykP8kKpghz6
  • QmecPw1iYuu85rtdYL2J2W9qcr6p8ijich9P5GbEAmmbW5
  • Qmaz1R8vcv9v3gUfksqiS9JUz7K9G8S5By3JYn8kTiiP5K

  In the case you're indexing one of the impacted subgraphs, you should first upgrade the graph-node version, then rewind the affected subgraphs to the smallest startBlock of their subgraph manifest. To achieve that, you can use the following graphman rewind CLI command invocations:

  graphman rewind 0x8aed5a76735b0890d4c0a9e752c88cba4f3d9d0e4e1dad70d6bd913cc279e4b8 7913573 QmNY7gDNXHECV8SXoEY7hbfg4BX1aDMxTBDiFuG4huaSGA
  graphman rewind 0x15da49d2177dde3012b67f12d25d9ae4cb56c3638c2ec011298e509be27be77a 7303699 QmYzsCjrVwwXtdsNm3PZVNziLGmb9o513GUzkq5wwhgXDT
  graphman rewind 0x1be8b52bd6b3c29b1ead63f6a8bef83d626d8b6795f6c2d89c302cbe32d59830 10613640 QmYUcrn9S1cuSZQGomLRyn8GbNHmX8viqxMykP8kKpghz6
  graphman rewind 0xef547ca70b451cc9fedbc039edcfd4a35de80fce76ed6ad31bc15a47850dfefd 10736242 QmecPw1iYuu85rtdYL2J2W9qcr6p8ijich9P5GbEAmmbW5
  graphman rewind 0x8fb3744b077831856203bbfe465e9cb681d0ed0c67db297faee54f0542f045d3 5533890 Qmaz1R8vcv9v3gUfksqiS9JUz7K9G8S5By3JYn8kTiiP5K
  
  # Note: the following subgraph (https://github.com/Data-Nexus/721-Marketplace) is extremely
  # write-heavy, and your node may OOM while rewinding. If you run into this issue suggest
  # you either rewind in smaller chunks of blocks, or simply redeploy it.
  graphman rewind 0xf9bc13c7864c997aaed501715a0627a25e748c2df4889cc9cbf4d077e7bca34a 4944642 QmccAwofKfT9t4XKieDqwZre1UUZxuHw5ynB35BHwHAJDT
  

  See #4149 for more information.

What's new

• Grafted subgraphs can now add their own data sources. #3989, #4027, #4030
• Add support for filtering by nested interfaces. #3677
• Add support for message handlers in Cosmos #3975
• Dynamic data sources for Firehose-backed subgraphs. #4075
• Various logging improvements. #4078, #4084, #4031, #4144, #3990
• Some DB queries now have GCP Cloud Insight -compliant tags that show where the query originated from. #4079
• New configuration variable GRAPH_STATIC_FILTERS_THRESHOLD to conditionally enable static filtering based on the number of dynamic data sources. #4008
• New configuration variable GRAPH_STORE_BATCH_TARGET_DURATION. #4133

Docker image

• The official Docker image now runs on Debian 11 "Bullseye". #4081
• We now ship envsubst with the official Docker image, allowing you to easily run templating logic on your configuration files. #3974

Graphman

We have a new documentation page for graphman, check it out here!

• Subgraph pruning with graphman! #3898, #4125, #4153, #4152, #4156, #4041
• New command graphman drop to hastily delete a subgraph deployment. #4035
• New command graphman chain call-cache for clearing the call cache for a given chain. #4066
• Add --delete-duplicates flag to graphman check-blocks by @tilacog in #3988

Performance

• Restarting a node now takes much less time because postgres_fdw user mappings are only rebuilt upon schema changes. If necessary, you can also use the new commands graphman database migrate and graphman database remap to respectively apply schema migrations or run remappings manually. #4009, #4076
• Database replicas now won't fall behind as much when copying subgraph data. #3966 #3986
• Block handlers optimization with Firehose >= 1.1.0. #3971
• Reduced the amount of data that a non-primary shard has to mirror from the primary shard. #4015
• We now use advisory locks to lock deployments' tables against concurrent writes. #4010

Bug fixes

• Fixed a bug that would cause some failed subgraphs to never restart. #3959
• Fixed a bug that would cause bad POIs for Firehose-backed subgraphs when processing CREATE calls. #4085
• Fixed a bug which would cause failure to redeploy a subgraph immediately after deletion. #4044
• Firehose connections are now load-balanced. #4083
• Determinism fixes. See above. #4055, #4149

Dependency updates

Dependency	updated to
anyhow	1.0.66
base64	0.13.1
clap	3.2.23
env_logger	0.9.1
iana-time-zone	0.1.47
itertools	0.10.5
jsonrpsee	0.15.1
num_cpus	1.14.0
openssl	0.10.42
pretty_assertions	1.3.0
proc-macro2	1.0.47
prometheus	0.13.3
protobuf-parse	3.2.0
semver	1.0.14
serde_plain	1.0.1
sha2	0.10.6
structopt	removed
tokio-stream	0.1.11
tokio-tungstenite	0.17.2
tower-test	d27ba65
url	2.3.1

v0.29.0-rc.0

v0.29.0-rc.0 is the first release candidate for the upcoming v0.29.0. This release candidate is intended for testnet use only.

$ docker pull graphprotocol/graph-node:v0.29.0-rc.0

Edit 2022-12-06: v0.29.0-rc.0 has been promoted to v0.29.0.

v0.28.2

Indexers are advised to migrate to v0.28.2 and entirely bypass v0.28.0 and v0.28.1.

Fixed a bug which would cause subgraphs to stop syncing under some graph-node deployment configurations. #4046, #4051

v0.28.1

Yanked. Please migrate to v0.28.2.

v0.28.0

This is the v0.28.0 release.

Upgrade notes

• New DB table for dynamic data sources.
  For new subgraph deployments, dynamic data sources will be recorded under the sgd*.data_sources$ table, rather than subgraphs.dynamic_ethereum_contract_data_source. As a consequence new deployments will not work correctly on earlier graph node versions, so downgrading to an earlier graph node version is not supported.
  See issue #3405 for other details.

What's new

• The filepath which "too expensive qeueries" are sourced from is now configurable. You can use either the GRAPH_NODE_EXPENSIVE_QUERIES_FILE environment variable or the expensive_queries_filename option in the TOML configuration. #3710
• The output you'll get from graphman query is less cluttered and overall nicer. The new options --output and --trace are available for detailed query information. #3860
• docker build will now --target the production build stage by default. When you want to get the debug build, you now need --target graph-node-debug. #3814
• Node IDs can now contain any character. The Docker start script still replaces hyphens with underscores for backwards compatibility reasons, but this behavior can be changed with the GRAPH_NODE_ID_USE_LITERAL_VALUE environment variable. With this new option, you can now seamlessly use the K8s-provided host names as node IDs, provided you reassign your deployments accordingly. #3688
• You can now use the conn_pool_size option in TOML configuration files to configure the connection pool size for Firehose providers. #3833
• Index nodes now have an endpoint to perform block number to canonical hash conversion, which will unblock further work towards multichain support. #3942
• _meta.block.timestamp is now available for subgraphs indexing EVM chains. #3738, #3902
• The deployment_eth_rpc_request_duration metric now also observes eth_getTransactionReceipt requests' duration. #3903
• New Prometheus metrics query_parsing_time and query_validation_time for monitoring query processing performance. #3760
• New command graphman config provider, which shows what providers are available for new deployments on a given network and node. #3816
  E.g. $ graphman --node-id index_node_0 --config graph-node.toml config provider mainnet
• Experimental support for GraphQL API versioning has landed. #3185
• Progress towards experimental support for off-chain data sources. #3791
• Experimental integration for substreams. #3777, #3784, #3897, #3765, and others

Bug fixes

• graphman stats now complains instead of failing silently when incorrectly setting account-like optimizations. #3918
• Fixed inconsistent logic in the provider selection when the limit TOML configuration option was set. #3816
• Fixed issues that would arise from dynamic data sources' names clashing against template names. #3851
• Dynamic data sources triggers are now processed by insertion order. #3851, #3854
• When starting, the Docker image now replaces the bash process with the graph-node process (with a PID of 1). #3803
• Refactor subgraph store tests by @evaporei in #3662
• The ethereum_chain_head_number metric doesn't get out of sync anymore on chains that use Firehose. #3771, #3732
• Fixed a crash caused by bad block data from the provider. #3944
• Fixed some minor Firehose connectivity issues via TCP keepalive, connection and request timeouts, and connection window size tweaks. #3822, #3855, #3877, #3810, #3818
• Copying private data sources' tables across shards now works as expected. #3836

Performance improvements

• Firehose GRPC stream requests are now compressed with gzip, if the server supports it. #3893
• Memory efficiency improvements within the entity cache. #3594
• Identical queries now benefit from GraphQL validation caching, and responses are served faster. #3759

Other

• Avoid leaking some sensitive information in logs. #3812

v0.27.0

• Store writes are now carried out in parallel to the rest of the subgraph process, improving indexing performance for subgraphs with significant store interaction. Metrics & monitoring was updated for this new pipelined process;
• This adds support for apiVersion 0.0.7, which makes receipts accessible in Ethereum event handlers. Documentation link;
• This introduces some improvements to the subgraph GraphQL API, which now supports filtering on the basis of, and filtering for entities which changed from a certain block;
• Support was added for Arweave indexing. Tendermint was renamed to Cosmos in Graph Node. These integrations are still in "beta";
• Callhandler block filtering for contract calls now works as intended (this was a longstanding bug);
• Gas costing for mappings is still set at a very high default, as we continue to benchmark and refine this metric;
• A new graphman fix block command was added to easily refresh a block in the block cache, or clear the cache for a given network;
• IPFS file fetching now uses files/stat, as object was deprecated;
• Subgraphs indexing via a Firehose can now take advantage of Firehose-side filtering;
• NEAR subgraphs can now match accounts for receipt filtering via prefixes or suffixes.

Upgrade notes

• In the case of you having custom SQL, there's a new SQL migration;
• On the pipelining of the store writes, there's now a new environment variable GRAPH_STORE_WRITE_QUEUE (default value is 5), that if set to 0, the old synchronous behaviour will come in instead. The value stands for the amount of write/revert parallel operations #3177;
• There's now support for TLS connections in the PostgreSQL notification_listener #3503;
• GraphQL HTTP and WebSocket ports can now be set via environment variables #2832;
• The genesis block can be set via the GRAPH_ETHEREUM_GENESIS_BLOCK_NUMBER env var #3650;
• There's a new experimental feature to limit the number of subgraphs for a specific web3 provider. Link for documentation;
• Two new GraphQL validation environment variables were included: ENABLE_GRAPHQL_VALIDATIONS and SILENT_GRAPHQL_VALIDATIONS, which are documented here;
• A bug fix for graphman index was landed, which fixed the behavior where if one deployment was used by multiple names would result in the command not working #3416;
• Another fix landed for graphman, the bug would allow the unassign/reassign commands to make two or more nodes index the same subgraph by mistake #3478;
• Error messages of eth RPC providers should be clearer during graph-node start up #3422;
• Env var GRAPH_STORE_CONNECTION_MIN_IDLE will no longer panic, instead it will log a warning if it exceeds the pool_size #3489;
• Failed GraphQL queries now have proper timing information in the service metrics #3508;
• Non-primary shards now can be disabled through setting the pool_size to 0 #3513;
• Queries with large results now have a query_id #3514;
• It's now possible to disable the LFU Cache by setting GRAPH_QUERY_LFU_CACHE_SHARDS to 0 #3522;
• GRAPH_ACCOUNT_TABLES env var is not supported anymore #3525;
• New documentation landed on the metadata tables;
• GRAPH_GRAPHQL_MAX_OPERATIONS_PER_CONNECTION for GraphQL subscriptions now has a default of 1000 #3735

v0.26.0

0.26.0

Features

• Gas metering #2414
• Adds support for Solidity Custom Errors #2577
• Debug fork tool #2995 #3292
• Automatically remove unused deployments #3023
• Fix fulltextsearch space handling #3048
• Allow placing new deployments onto one of several shards #3049
• Make NEAR subgraphs update their sync status #3108
• GraphQL validations #3164
• Add special treatment for immutable entities #3201
• Tendermint integration #3212
• Skip block updates when triggers are empty #3223 #3268
• Use new GraphiQL version #3252
• GraphQL prefetching #3256
• Allow using Bytes as well as String/ID for the id of entities #3271
• GraphQL route for dumping entity changes in subgraph and block #3275
• Firehose filters #3323
• NEAR filters #3372

Robustness

• Improve our CacheWeight estimates #2935
• Refactor GraphQL execution #3005
• Setup databases in parallel #3019
• Block ingestor now fetches receipts in parallel #3030
• Prevent subscriptions from back-pressuring the notification queue #3053
• Avoid parsing X triggers if the filter is empty #3083
• Pipeline BlockStream #3085
• More robust proofOfIndexing GraphQL route #3348

graphman

• Add run command, for running a subgraph up to a block #3079
• Add analyze command, for analyzing a PostgreSQL table, which can improve performance #3170
• Add index create command, for adding an index to certain attributes #3175
• Add index list command, for listing indexes #3198
• Add index drop command, for dropping indexes #3198

Dependency Updates

These are the main ones:

• Updated protobuf to latest version for NEAR #2947
• Update web3 crate #2916 #3120 #3338
• Update graphql-parser to v0.4.0 #3020
• Bump itertools from 0.10.1 to 0.10.3 #3037
• Bump clap from 2.33.3 to 2.34.0 #3039
• Bump serde_yaml from 0.8.21 to 0.8.23 #3065
• Bump tokio from 1.14.0 to 1.15.0 #3092
• Bump indexmap from 1.7.0 to 1.8.0 #3143
• Update ethabi to its latest version #3144
• Bump structopt from 0.3.25 to 0.3.26 #3180
• Bump anyhow from 1.0.45 to 1.0.53 #3182
• Bump quote from 1.0.9 to 1.0.16 #3112 #3183 #3384
• Bump tokio from 1.15.0 to 1.16.1 #3208
• Bump semver from 1.0.4 to 1.0.5 #3229
• Bump async-stream from 0.3.2 to 0.3.3 #3361
• Update jsonrpc-server #3313

Misc

• More context when logging RPC calls #3128
• Increase default reorg threshold to 250 for Ethereum #3308
• Improve traces error logs #3353
• Add warning and continue on parse input failures for Ethereum #3326

Upgrade Notes

When upgrading to this version, we recommend taking a brief look into these changes:

• Gas metering #2414
  • Now there's a gas limit for subgraph mappings, if the limit is reached the subgraph will fail with a non-deterministic error, you can make them recover via the environment variable GRAPH_MAX_GAS_PER_HANDLER
• Improve our CacheWeight estimates #2935
  • This is relevant because a couple of releases back we've added a limit for the memory size of a query result. That limit is based of the CacheWeight.

These are some of the features that will probably be helpful for indexers 😊

• Allow placing new deployments onto one of several shards #3049
• GraphQL route for dumping entity changes in subgraph and block #3275
• Unused deployments are automatically removed now #3023
  • The interval can be set via GRAPH_REMOVE_UNUSED_INTERVAL
• Setup databases in parallel #3019
• Block ingestor now fetches receipts in parallel #3030
  • GRAPH_ETHEREUM_FETCH_TXN_RECEIPTS_IN_BATCHES can be set to true for the old fetching behavior
• More robust proofOfIndexing GraphQL route #3348
  • A token can be set via GRAPH_POI_ACCESS_TOKEN to limit access to the POI route
• The new graphman commands 🙂

v0.25.2

This release includes two changes:

• Bug fix of blocks being skipped from processing when: a deterministic error happens and the index-node gets restarted. Issue #3236, Pull Request: #3316.
• Automatic retries for non-deterministic errors. Issue #2945, Pull Request: #2988.

This is the last patch on the 0.25 minor version, soon 0.26.0 will be released. While that we recommend updating to this version to avoid determinism issues that could be caused on graph-node restarts.

v0.25.1

This release only adds two fixes:

• The first is to address an issue with decoding the input of some calls #3194 where subgraphs that would try to index contracts related to those would fail. Now they can advance normally.
• The second one is to fix a non-determinism issue with the retry mechanism for errors. Whenever a non-deterministic error happened, we would keep retrying to process the block, however we should've clear the EntityCache on each run so that the error entity changes don't get transacted/saved in the database in the next run. This could make the POI generation non-deterministic for subgraphs that failed and retried for non-deterministic reasons, adding a new entry to the database for the POI.

We strongly recommend updating to this version as quickly as possible.

v0.25.0

Api Version 0.0.6

This release ships support for API version 0.0.6 in mappings:

• Added nonce field for Transaction objects.
• Added baseFeePerGas field for Block objects (EIP-1559).

Block Cache Invalidation and Reset

All cached block data must be refetched to account for the new Block and Trasaction
struct versions, so this release includes a graph-node startup check that will:

1. Truncate all block cache tables.
2. Bump the db_version value from 2 to 3.

(Table truncation is a fast operation and no downtime will occur because of that.)

Ethereum

• 'Out of gas' errors on contract calls are now considered deterministic errors,
  so they can be handled by try_ calls. The gas limit is 50 million.

Environment Variables

• The GRAPH_ETH_CALL_GAS environment is removed to prevent misuse, its value
  is now hardcoded to 50 million.

Multiblockchain

• Initial support for NEAR subgraphs.
• Added FirehoseBlockStream implementation of BlockStream (#2716)

Misc

• Rust docker image is now based on Debian Buster.
• Optimizations to the PostgreSQL notification queue.
• Improve PostgreSQL robustness in multi-sharded setups. (#2815)
• Added 'networks' to the 'subgraphFeatures' endpoint. (#2826)
• Check and limit the size of GraphQL query results. (#2845)
• Allow _in and _not_in GraphQL filters. (#2841)
• Add PoI for failed subgraphs. (#2748)
• Make graphman rewind safer to use. (#2879)
• Add subgraphErrors for all GraphQL schemas. (#2894)
• Add Graph-Attestable response header. (#2946)
• Add support for minimum block constraint in GraphQL queries (number_gte) (#2868).
• Handle revert cases from Hardhat and Ganache (#2984)
• Fix bug on experimental prefetching optimization feature (#2899)