GRPC stream - how to handle gaps in the GRPC stream

[MoonShiesty]

Describe your question in detail.

I received a batch from the aptos GRPC stream with a gap, causing the aptos-indexer-processor process to panic

if this is an unexepected error, i wanted to bring it to the attention to the aptos indexer team. otherwise, if this is an error that is expected to happen, is there a way to gracefully recover from a gap in the stream?

What error, if any, are you getting?

aptos-indexer-processor panicked with the following error:

{"timestamp":"2024-04-23T23:03:48.135862Z","level":"ERROR","fields":{"message":"details = """\npanicked at processor/src/grpc_stream.rs:370:21:\n[Parser] Received batch with gap from GRPC stream"""\nbacktrace = """\n 0: 0x556d8521ede8 - backtrace::backtrace::trace::he96c8a6374e0dec4\n 1: 0x556d8520079d - backtrace::capture::Backtrace::new::hda69e46c29033b0d\n 2: 0x556d84f4cf5e - server_framework::setup_panic_handler::{{closure}}::h8b1cbcf38cddfa79\n 3: 0x556d852d48d6 - std::panicking::rust_panic_with_hook::hb164d19c0c1e71d4\n 4: 0x556d852d45e9 - std::panicking::begin_panic_handler::{{closure}}::h0369088c533c20e9\n 5: 0x556d852d32a6 - std::sys_common::backtrace::__rust_end_short_backtrace::hc11d910daf35ac2e\n 6: 0x556d852d4374 - rust_begin_unwind\n 7: 0x556d840bc7c5 - core::panicking::panic_fmt::ha6effc2775a0749c\n 8: 0x556d845f4220 - processor::grpc_stream::create_fetcher_loop::{{closure}}::h7f803291d071375e\n 9: 0x556d84695955 - processor::worker::Worker::run::{{closure}}::{{closure}}::h61247439de121ed5\n 10: 0x556d845dd46f - tokio::runtime::task::core::Core<T,S>::poll::h36caa4003ea88b98\n 11: 0x556d84714901 - tokio::runtime::task::harness::Harness<T,S>::poll::h81410bd731a53628\n 12: 0x556d851c4da3 - tokio::runtime::scheduler::multi_thread::worker::Context::run_task::h7f3f5cc0f641d8c7\n 13: 0x556d851c3e77 - tokio::runtime::scheduler::multi_thread::worker::Context::run::he5653e5edfe1a67a\n 14: 0x556d851e59af - tokio::runtime::context::runtime::enter_runtime::hcf237e4b22099fe8\n 15: 0x556d851c30ac - tokio::runtime::scheduler::multi_thread::worker::run::h92ff26f9f5c5dedd\n 16: 0x556d851c61c3 - <tokio::runtime::blocking::task::BlockingTask as core::future::future::Future>::poll::h0136d5b83a079a88\n 17: 0x556d851c78c7 - tokio::runtime::task::core::Core<T,S>::poll::h08b74144ba2bd92a\n 18: 0x556d851b7d8a - tokio::runtime::task::harness::Harness<T,S>::poll::h66934e7c79bfae2f\n 19: 0x556d851cd690 - tokio::runtime::blocking::pool::Inner::run::h87295cf3798f6033\n 20: 0x556d851cea77 - std::sys_common::backtrace::__rust_begin_short_backtrace::h54039ba8b19b91a1\n 21: 0x556d851ba8b9 - core::ops::function::FnOnce::call_once{{vtable.shim}}::h1222293b78641478\n 22: 0x556d852d6e75 - std::sys::pal::unix::thread::Thread::new::thread_start::h3631815ad38387d6\n 23: 0x7f7c5ea56ac3 - start_thread\n at ./nptl/pthread_create.c:442:8\n 24: 0x7f7c5eae8850 - __GI___clone3\n at ./misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81\n 25: 0x0 - \n"""\n"},"filename":"server-framework/src/lib.rs","line_number":124,"threadName":"tokio-runtime-worker","threadId":"ThreadId(3)"}

What have you tried or looked at? Or how can we reproduce the error?

i wasn't able to reproduce the issue after restarting the processor

Which operating system are you using?

Linux (Ubuntu, Fedora, Windows WSL, etc.)

Which SDK or tool are you using? (if any)

Indexer API

Describe your environment or tooling in detail

im running a custom aptos indexer processor on testnet, based on a recent aptos-indexer-processor (d475078299c6d53b69ff460cbf839d990291e2f7), virtualized ubuntu 22.04 running on an epyc chip. ive implemented a custom processor but haven't changed any other aptos-indexer-processor code

my processor is running on a 2g network connection with 128 threads, processing around 13 tps per-thread

[larry-aptos]

do you have logs before this line? if not, do you know at what version it breaks?