Indexer Pipeline Fails Under Heavy Traffic — Log Stream Stuck (v0.8.2)

[azharprabudi]

Hello everyone,

Is anyone else experiencing the same issue with Quickwit running on version 0.8.2?

Error log:

2025-06-24T12:56:33.488Z ERROR quickwit_indexing::actors::indexing_pipeline: Indexing pipeline failure. 
pipeline_id=IndexingPipelineId { node_id: "xxxxxxxxxxxx", index_uid: IndexUid { index_id: "xxxxxxxxxxxx", incarnation_id: Ulid(2111614674913088029481757961785957061) }, source_id: "xxxxxxxxxxxx", pipeline_uid: Pipeline(01JYH04PBD671DDN9ERS4GC4JT) } 
generation=1 
healthy_actors=["quickwit_indexing::actors::index_serializer::IndexSerializer-divine-rkWv"] 
failed_or_unhealthy_actors=["SourceActor-polished-WOIC", "quickwit_indexing::actors::doc_processor::DocProcessor-winter-injJ", "Indexer-wild-jV9A"] 
success_actors=["Packager-purple-cIqb", "IndexUploader-ancient-Ppq1", "quickwit_indexing::actors::sequencer::Sequencer<quickwit_indexing::actors::publisher::Publisher>-late-4Rw0", "Publisher-cold-VSE0"]

I’m using Kafka as the source. Under normal traffic conditions, everything works fine. However, during heavy traffic or when there’s an external security scan, the Quickwit indexer pipeline encounters errors and fails to consume data from Kafka, which causes the log stream to become stuck.

I’ve tried skipping offsets from the Kafka consumer group, but it didn’t work. From what I’ve read in other GitHub issues, it seems there’s no built-in way to skip Kafka offsets in Quickwit.

Questions:

Is it true that there’s currently no way to skip Kafka offsets in Quickwit?

Is there any recommended approach or best practice to prevent the logs from getting stuck in this scenario?

Any advice or workaround would be greatly appreciated.

Thank you!

[rdettai]

In your description you mention an issue with heavy load, but it seems that you have identified that the problem is actually a document or a batch of documents that systematically crash the indexing pipeline. Am I correct?

Is it true that there’s currently no way to skip Kafka offsets in Quickwit?

Not through the API, but you can modify the index_metadata directly in the metastore to move the checkpoint forward (which obviously creates data loss).

Is there any recommended approach or best practice to prevent the logs from getting stuck in this scenario?

In general, the indexing pipeline should not fail because of bad documents. Do you have any panic or error message prior to the indexing pipeline failure that would suggest the reason?

[azharprabudi]

Hi @rdettai, really appreciate it your response

In your description you mention an issue with heavy load, but it seems that you have identified that the problem is actually a document or a batch of documents that systematically crash the indexing pipeline. Am I correct?

Suppose to be yes, because in the daily normal operation, even if we have some heavy traffics, the logs are normal, and can be ingested to QuickWit, but sometimes, when there's a scanning from the outside, it makes the quickwit indexing pipelines are crashed

Not through the API, but you can modify the index_metadata directly in the metastore to move the checkpoint forward (which obviously creates data loss).

I think it will be fine as long as the indexing can back running properly, may i know for this thing, do i need to update the data on database side or is there an api for updating the index metadata in the metastore ?

In general, the indexing pipeline should not fail because of bad documents. Do you have any panic or error message prior to the indexing pipeline failure that would suggest the reason?

Sure here's the panic logs

The term has an invalid type code stack backtrace: 0: rust_begin_unwind at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:645:5 1: core::panicking::panic_fmt at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/panicking.rs:72:14 2: core::panicking::panic_display at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/panicking.rs:196:5 3: core::panicking::panic_str at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/panicking.rs:171:5 4: core::option::expect_failed at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/option.rs:1980:5 5: <tantivy::postings::json_postings_writer::JsonPostingsWriter<Rec> as tantivy::postings::postings_writer::PostingsWriter>::serialize 6: tantivy::postings::postings_writer::serialize_postings 7: tantivy::indexer::segment_writer::SegmentWriter::finalize 8: tantivy::indexer::single_segment_index_writer::SingleSegmentIndexWriter<D>::finalize 9: quickwit_indexing::models::indexed_split::IndexedSplitBuilder::finalize 10: <quickwit_indexing::actors::index_serializer::IndexSerializer as quickwit_actors::actor::Handler<quickwit_indexing::models::indexed_split::IndexedSplitBatchBuilder>>::handle::{{closure}} 11: <H as quickwit_actors::actor::DeferableReplyHandler<M>>::handle_message::{{closure}} 12: <core::option::Option<(tokio::sync::oneshot::Sender<<A as quickwit_actors::actor::DeferableReplyHandler<M>>::Reply>,M)> as quickwit_actors::envelope::EnvelopeT<A>>::handle_message::{{closure}} 13: quickwit_actors::spawn_builder::ActorExecutionEnv<A>::process_one_message::{{closure}} 14: quickwit_actors::spawn_builder::SpawnBuilder<A>::spawn::{{closure}} 15: tokio::runtime::task::core::Core<T,S>::poll 16: tokio::runtime::task::harness::Harness<T,S>::poll 17: tokio::runtime::scheduler::multi_thread::worker::Context::run_task 18: tokio::runtime::scheduler::multi_thread::worker::Context::run 19: tokio::runtime::context::scoped::Scoped<T>::set 20: tokio::runtime::context::runtime::enter_runtime 21: tokio::runtime::scheduler::multi_thread::worker::run 22: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll 23: tokio::runtime::task::core::Core<T,S>::poll 24: tokio::runtime::task::harness::Harness<T,S>::poll note: Some details are omitted, run with RUST_BACKTRACE=full for a verbose backtrace. 2025-06-24T12:51:18.151Z ERROR quickwit_actors::spawn_builder: actor-failure cause=the channel is closed

[fulmicoton-dd]

just replacing the image tag should do the job on a minor.

[fulmicoton-dd]

If the problem persists after the upgrade, let us know. We will work on an image to identify the exact faulty document and the associated bugs.

[azharprabudi]

ok i'll try on the development environment first, thanks everyone, appreciate it

[PSeitz-dd]

0.8.2 does not include the bugfix quickwit-oss/tantivy#2445, so upgrading should fix the issue for you.

[azharprabudi]

Appreciate it @PSeitz-dd, I have upgraded my Quickwit clusters for all environments, I'll monitor in next few days