Skip to content

mongodb/cdc: make progress on low activity collections #3430

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 12 commits into from
May 27, 2025
+252 −126
Merged

mongodb/cdc: make progress on low activity collections #3430

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
43c5b49
chore: bump mongodb version
rockwotj May 22, 2025
4b3e063
mongodb/cdc: checkpoint when completing snapshot
rockwotj May 22, 2025
d54f530
mongodb/cdc: fix busted integration_test
rockwotj May 22, 2025
2f72435
mongodb/cdc: fix wierd driver connection timeout bug
rockwotj May 22, 2025
0242c4e
mongodb/cdc: simplify timeout workaround
rockwotj May 22, 2025
bfe63bc
mongodb/cdc: demonstrate better checkpointing when no changes
rockwotj May 22, 2025
e7b0fd8
chore: upgrade testcontainers and checkpoint
rockwotj May 23, 2025
4a2a611
mongodb/cdc: simplify checkpointing
rockwotj May 23, 2025
727db46
chore: make deps
rockwotj May 23, 2025
f52f875
ollama: fix test
rockwotj May 23, 2025
b72e342
chore: update changelog
rockwotj May 23, 2025
a426e7c
chore: make linter happy
rockwotj May 23, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ All notable changes to this project will be documented in this file.
### Fixed

- Fixed an issue where the `aws_kinesis` input would cause high CPU utilization in cases where a shard has a trickle of data and a batching period is specified.
- Fixed an issue where the `mongodb_cdc` inputs could have spurious errors when collections had no writes for > 30 seconds. (@rockwotj)

## 4.55.1 - 2025-05-19

Expand Down
60 changes: 31 additions & 29 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ require (
github.com/ClickHouse/clickhouse-go/v2 v2.34.0
github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/trace v1.24.1
github.com/IBM/sarama v1.43.3
github.com/Jeffail/checkpoint v1.0.1
github.com/Jeffail/checkpoint v1.1.0
github.com/Jeffail/gabs/v2 v2.7.0
github.com/Jeffail/shutdown v1.0.0
github.com/Masterminds/semver v1.5.0
Expand Down Expand Up @@ -131,8 +131,8 @@ require (
github.com/snowflakedb/gosnowflake v1.11.0
github.com/sourcegraph/conc v0.3.0
github.com/stretchr/testify v1.10.0
github.com/testcontainers/testcontainers-go/modules/ollama v0.32.0
github.com/testcontainers/testcontainers-go/modules/qdrant v0.32.0
github.com/testcontainers/testcontainers-go/modules/ollama v0.37.0
github.com/testcontainers/testcontainers-go/modules/qdrant v0.37.0
github.com/tetratelabs/wazero v1.7.3
github.com/timeplus-io/proton-go-driver/v2 v2.0.17
github.com/tmc/langchaingo v0.1.13
Expand All @@ -148,15 +148,15 @@ require (
github.com/xitongsys/parquet-go v1.6.2
github.com/xitongsys/parquet-go-source v0.0.0-20211228015320-b4f792c43cd0
github.com/youmark/pkcs8 v0.0.0-20240726163527-a2c0da244d78
go.mongodb.org/mongo-driver/v2 v2.1.0
go.mongodb.org/mongo-driver/v2 v2.2.1
go.nanomsg.org/mangos/v3 v3.4.2
go.opentelemetry.io/otel v1.35.0
go.opentelemetry.io/otel v1.36.0
go.opentelemetry.io/otel/exporters/jaeger v1.17.0
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.28.0
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.31.0
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.28.0
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.28.0
go.opentelemetry.io/otel/sdk v1.35.0
go.opentelemetry.io/otel/trace v1.35.0
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.31.0
go.opentelemetry.io/otel/sdk v1.36.0
go.opentelemetry.io/otel/trace v1.36.0
go.starlark.net v0.0.0-20250318223901-d9371fef63fe
go.uber.org/multierr v1.11.0
golang.org/x/crypto v0.38.0
Expand Down Expand Up @@ -186,8 +186,9 @@ require (
github.com/certifi/gocertifi v0.0.0-20210507211836-431795d63e8d // indirect
github.com/cloudflare/circl v1.6.0 // indirect
github.com/cncf/xds/go v0.0.0-20250121191232-2f005788dc42 // indirect
github.com/containerd/platforms v0.2.1 // indirect
github.com/containerd/platforms v1.0.0-rc.1 // indirect
github.com/cyphar/filepath-securejoin v0.4.1 // indirect
github.com/ebitengine/purego v0.8.4 // indirect
github.com/emirpasic/gods v1.18.1 // indirect
github.com/envoyproxy/go-control-plane/envoy v1.32.4 // indirect
github.com/envoyproxy/protoc-gen-validate v1.2.1 // indirect
Expand All @@ -199,6 +200,8 @@ require (
github.com/jzelinskie/stringz v0.0.3 // indirect
github.com/kevinburke/ssh_config v1.2.0 // indirect
github.com/mitchellh/mapstructure v1.5.0 // indirect
github.com/moby/go-archive v0.1.0 // indirect
github.com/moby/sys/atomicwriter v0.1.0 // indirect
github.com/moby/sys/userns v0.1.0 // indirect
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
github.com/modern-go/reflect2 v1.0.2 // indirect
Expand All @@ -212,6 +215,7 @@ require (
github.com/planetscale/vtprotobuf v0.6.1-0.20240319094008-0393e58bdf10 // indirect
github.com/samber/lo v1.47.0 // indirect
github.com/sergi/go-diff v1.3.2-0.20230802210424-5b0b94c5c0d3 // indirect
github.com/shirou/gopsutil/v4 v4.25.4 // indirect
github.com/siddontang/go v0.0.0-20180604090527-bdc77568d726 // indirect
github.com/siddontang/go-log v0.0.0-20180807004314-8d05993dda07 // indirect
github.com/skeema/knownhosts v1.3.1 // indirect
Expand All @@ -226,7 +230,7 @@ require (
gitlab.com/golang-commonmark/puny v0.0.0-20191124015043-9f83538fa04f // indirect
go.opentelemetry.io/auto/sdk v1.1.0 // indirect
go.opentelemetry.io/contrib/detectors/gcp v1.35.0 // indirect
go.opentelemetry.io/otel/sdk/metric v1.35.0 // indirect
go.opentelemetry.io/otel/sdk/metric v1.36.0 // indirect
gonum.org/v1/gonum v0.15.1 // indirect
gopkg.in/go-jose/go-jose.v2 v2.6.3 // indirect
gopkg.in/warnings.v0 v0.1.2 // indirect
Expand All @@ -240,14 +244,14 @@ require (
cloud.google.com/go/iam v1.4.2 // indirect
cloud.google.com/go/trace v1.11.3 // indirect
cuelang.org/go v0.12.1 // indirect
dario.cat/mergo v1.0.0 // indirect
dario.cat/mergo v1.0.2 // indirect
filippo.io/edwards25519 v1.1.0 // indirect
github.com/99designs/go-keychain v0.0.0-20191008050251-8e49817e8af4 // indirect
github.com/99designs/keyring v1.2.2 // indirect
github.com/AthenZ/athenz v1.10.43 // indirect
github.com/Azure/azure-sdk-for-go v68.0.0+incompatible // indirect
github.com/Azure/azure-sdk-for-go/sdk/internal v1.10.0 // indirect
github.com/Azure/go-ansiterm v0.0.0-20230124172434-306776ec8161 // indirect
github.com/Azure/go-ansiterm v0.0.0-20250102033503-faa5f7b0171c // indirect
github.com/AzureAD/microsoft-authentication-library-for-go v1.3.3 // indirect
github.com/ClickHouse/ch-go v0.65.1 // indirect
github.com/DataDog/zstd v1.5.2 // indirect
Expand Down Expand Up @@ -307,7 +311,7 @@ require (
github.com/distribution/reference v0.6.0 // indirect
github.com/dlclark/regexp2 v1.11.5 // indirect
github.com/docker/cli v26.1.4+incompatible // indirect
github.com/docker/docker v28.0.4+incompatible // indirect
github.com/docker/docker v28.1.1+incompatible // indirect
github.com/docker/go-connections v0.5.0 // indirect
github.com/docker/go-units v0.5.0 // indirect
github.com/dvsekhvalnov/jose2go v1.6.0 // indirect
Expand All @@ -331,7 +335,7 @@ require (
github.com/golang-sql/sqlexp v0.1.0 // indirect
github.com/golang/groupcache v0.0.0-20241129210726-2c02b8208cf8 // indirect
github.com/golang/protobuf v1.5.4 // indirect
github.com/golang/snappy v0.0.4 // indirect
github.com/golang/snappy v1.0.0 // indirect
github.com/google/flatbuffers v24.3.25+incompatible // indirect
github.com/google/pprof v0.0.0-20240827171923-fa2c70bbbfe5 // indirect
github.com/google/s2a-go v0.1.9 // indirect
Expand Down Expand Up @@ -380,16 +384,16 @@ require (
github.com/kylelemons/godebug v1.1.0 // indirect
github.com/lann/builder v0.0.0-20180802200727-47ae307949d0 // indirect
github.com/lann/ps v0.0.0-20150810152359-62de8c46ede0 // indirect
github.com/lufia/plan9stats v0.0.0-20240226150601-1dcf7310316a // indirect
github.com/magiconair/properties v1.8.7 // indirect
github.com/lufia/plan9stats v0.0.0-20250317134145-8bc96cf8fc35 // indirect
github.com/magiconair/properties v1.8.10 // indirect
github.com/mattn/go-colorable v0.1.14 // indirect
github.com/mattn/go-isatty v0.0.20 // indirect
github.com/mattn/go-runewidth v0.0.16 // indirect
github.com/moby/docker-image-spec v1.3.1 // indirect
github.com/moby/patternmatcher v0.6.0 // indirect
github.com/moby/sys/sequential v0.5.0 // indirect
github.com/moby/sys/user v0.1.0 // indirect
github.com/moby/term v0.5.0 // indirect
github.com/moby/sys/sequential v0.6.0 // indirect
github.com/moby/sys/user v0.4.0 // indirect
github.com/moby/term v0.5.2 // indirect
github.com/morikuni/aec v1.0.0 // indirect
github.com/mtibben/percent v0.2.1 // indirect
github.com/nats-io/nats-server/v2 v2.9.23 // indirect
Expand All @@ -399,7 +403,7 @@ require (
github.com/oapi-codegen/runtime v1.1.1 // indirect
github.com/olekukonko/tablewriter v0.0.5 // indirect
github.com/opencontainers/go-digest v1.0.0 // indirect
github.com/opencontainers/image-spec v1.1.0 // indirect
github.com/opencontainers/image-spec v1.1.1 // indirect
github.com/opencontainers/runc v1.1.13 // indirect
github.com/oschwald/maxminddb-golang v1.13.0 // indirect
github.com/paulmach/orb v0.11.1 // indirect
Expand All @@ -422,17 +426,15 @@ require (
github.com/segmentio/asm v1.2.0 // indirect
github.com/segmentio/encoding v0.4.0
github.com/segmentio/ksuid v1.0.4 // indirect
github.com/shirou/gopsutil/v3 v3.24.5 // indirect
github.com/shoenig/go-m1cpu v0.1.6 // indirect
github.com/shopspring/decimal v1.4.0 // indirect
github.com/sirupsen/logrus v1.9.3 // indirect
github.com/spaolacci/murmur3 v1.1.0 // indirect
github.com/stretchr/objx v0.5.2 // indirect
github.com/testcontainers/testcontainers-go v0.35.0
github.com/testcontainers/testcontainers-go/modules/mongodb v0.35.0
github.com/testcontainers/testcontainers-go v0.37.0 // indirect
github.com/testcontainers/testcontainers-go/modules/mongodb v0.37.0
github.com/tilinna/z85 v1.0.0 // indirect
github.com/tklauser/go-sysconf v0.3.13 // indirect
github.com/tklauser/numcpus v0.7.0 // indirect
github.com/tklauser/go-sysconf v0.3.15 // indirect
github.com/tklauser/numcpus v0.10.0 // indirect
github.com/urfave/cli/v2 v2.27.6
github.com/vmihailenco/tagparser/v2 v2.0.0 // indirect
github.com/xdg-go/pbkdf2 v1.0.0 // indirect
Expand All @@ -445,8 +447,8 @@ require (
gitlab.com/golang-commonmark/markdown v0.0.0-20211110145824-bf3e522c626a // indirect
go.opencensus.io v0.24.0 // indirect
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.60.0 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.60.0 // indirect
go.opentelemetry.io/otel/metric v1.35.0 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.61.0 // indirect
go.opentelemetry.io/otel/metric v1.36.0 // indirect
go.opentelemetry.io/proto/otlp v1.3.1 // indirect
go.uber.org/atomic v1.11.0 // indirect
go.uber.org/zap v1.27.0 // indirect
Expand Down
Loading