v1.21.3

Dragonfly v1.21.3

This is a patch release.

fix: display real replica remote address in INFO logs
fix: crash with concurrent multi-shard BLMOVE operations #3540

This follows up on Dragonfly v1.21.2 release.

v1.21.2

Dragonfly v1.21.2

This is a patch release.

fix: disable code that can cause potential deadlocks during the replication (#3521)

This follows up on Dragonfly v1.21.0 release, that includes the following prominent features:

• Alpha release of SSD Data tiering - enabled with flag --tiered_prefix some/path/basename
• Very basic support of multi-tenancy #3260
• HSETEX now supports NX option, see our docs.
• Added support for JSON.MERGE and for JSON.MSET.
• valkey's replica-announce-ip and --cluster--announc--ip are consolidated via --announce-ip flag.

Full Changelog: v1.21.1...v1.21.2

v1.21.1

Dragonfly v1.21.1

This is a patch release.

fix: the replication from older masters to newer versions (#3473)

This follows up on Dragonfly v1.21.0 release, that includes the following prominent features:

• Alpha release of SSD Data tiering - enabled with flag --tiered_prefix some/path/basename
• Very basic support of multi-tenancy @3260
• HSETEX now supports NX option, see our docs.
• Added support for JSON.MERGE and for JSON.MSET.
• valkey's replica-announce-ip and --cluster--announc--ip are consolidated via --announce-ip flag.

v1.21.0

Dragonfly v1.21.0

Some prominent changes include:

• Alpha release of SSD Data tiering - enabled with flag --prefix some/path/basename
• Very basic support of multi-tenancy @3260
• HSETEX now supports NX option, see our docs.
• Added support for JSON.MERGE and for JSON.MSET.
• valkey's replica-announce-ip and --cluster--announc--ip are consolidated via --announce-ip flag.

What's Changed

• feat(server): master stop sending exec opcode to replica by @adiholden in #3289
• chore: optimize zpopminmax operations by @romange in #3291
• fix(json_family): Fix error in JsonFamilyTest.MGet by @BagritsevichStepan in #3285
• fix: define macro WITH_AWS in cmake when flag is ON by @kostasrim in #3296
• chore: Add 'memory arena show' command by @romange in #3298
• fix: missing logs on pytest failures #3255 by @BorysTheDev in #3272
• chore: refactor compact_object and introduce materialize method by @romange in #3300
• fix(acl): loading interleaved plain and hashed passwords by @kostasrim in #3297
• chore: Add CompactObj Raw methods by @romange in #3303
• chore: On invalid TTL, print the TTL first by @chakaz in #3306
• chore: moving functions + renaming in tiered storage by @romange in #3304
• fix(tests): Add missing awaits by @chakaz in #3273
• chore: refactoring around tiered storage by @romange in #3299
• feat: yield when serialization is in progress by @kostasrim in #3220
• refactor: Use DbContext, OpArgs and Transaction to access DbSlice by @chakaz in #3311
• chore: Separate tiered serialization format from object values by @romange in #3310
• chore: add more community links to README by @Niennienzz in #3308
• fix(cluster): Join on specified attempt id by @chakaz in #3305
• chore(acl): add test with requirepass and aclfile by @kostasrim in #3312
• chore: skip test_cluster_flushall_during_migration by @kostasrim in #3316
• fix (pytest): generate unique random dbfilename for tests by @adiholden in #3317
• chore: bypass decoding/encoding of the data when performing offloading by @romange in #3315
• refactor: acl helpers and global tables by @kostasrim in #3309
• chore(tiering): Fixes by @dranikpg in #3225
• fix(migration): Use transactions! by @dranikpg in #3266
• fix: forbid DFLYCLUSTER commads set for emulated cluster mode by @BorysTheDev in #3307
• feat(namespaces): Initial support for multi-tenant by @chakaz in #3260
• chore: improve dfly_bench stats by @romange in #3320
• chore: remove replace_deleted flag from hnswlib by @dranikpg in #3323
• feat(test): Improve benchmark workflow by @adiholden in #3330
• fix: Proper shutdown sequence with Namespaces by @chakaz in #3333
• fix(test): copy logs for failed test during TEARDOWN phase by @BorysTheDev in #3331
• fix(json_family): fix JSON.STRAPPEND command for JSON legacy mode by @BagritsevichStepan in #3264
• chore(tiering): add protection against overruning memory budget by @romange in #3327
• chore: Add coordinated omission mode by @romange in #3332
• chore: implement sequential pass without the overlapping traffic by @romange in #3335
• feat: add an option to flush serialized entries on threshold limit by @kostasrim in #3241
• feat(hset_family): Add NX option to HSETEX by @BagritsevichStepan in #3295
• Fix blocking commands moved error by @BorysTheDev in #3334
• fix: ub in RegisterOnChange and regression tests for big values by @kostasrim in #3336
• fix: Cancel outgoing migration when retrying / closing by @chakaz in #3339
• chore: Make KeyIndex iterable by @dranikpg in #3326
• fix: AllocationTracker::Remove return value was reversed by @romange in #3341
• chore: remove redundant metrics from memory stats by @romange in #3345
• fix: corruption in replication stream by @romange in #3344
• chore: clean up TaskQueue since we do not need multiple fibers for it by @romange in #3348
• chore: pull helio by @romange in #3350
• chore: Log connection context when issuing dangerous cmds by @chakaz in #3352
• chore: small rename and add dcheck on LocalBlockingCounter by @kostasrim in #3356
• refactor: reduce number of logs for cluster by @BorysTheDev in #3357
• chore: fixes to dfly_bench by @romange in #3353
• chore: fix test_parser_memory_stats flakiness by @romange in #3354
• chore(server): Introduce StringSetWrapper by @dranikpg in #3347
• fix: do not upload offload values on a first hit by @romange in #3360
• chore(tiering): Range functions + small refactoring by @dranikpg in #3207
• fix: failure in test_cluster_fuzzymigration by @romange in #3363
• fix(server): Require >=1 args to GETEX by @chakaz in #3366
• fix(transaction): Fix namespace access by @dranikpg in #3364
• chore: disable compression on big values by @kostasrim in #3358
• fix: protect OnJournalEntry with ConditionGuard by @kostasrim in #3367
• chore: Introduce CoolQueue by @romange in #3365
• chore: small fixes around tiering by @romange in #3368
• chore: add a test for HeapSize() function by @romange in #3349
• chore: simplify computation of used_mem_current by @romange in #3372
• chore: disable cluster_fuzzymigration by @kostasrim in #3373
• chore(replica): remove unused methods in the Replica class by @BagritsevichStepan in #3374
• fix(transaction): Properly store block cancel status by @dranikpg in #3371
• chore: Track db_slice table memory instantly by @romange in #3375
• chore: add mem test for big values and default the flag by @kostasrim in #3369
• update: replication_acks_interval flag to 1000 by @kostasrim in #3378
• chore: dfly_bench - print ongoing error counts by @romange in #3382
• chore: introduce a cool queue that gradually retires cool items by @romange in #3377
• chore: update cached stats inside PollExecution by @romange in #3376
• fix: Fix test_take_over_seeder by @chakaz in #3385
• chore: reenable evictions upon insertion to avoid OOM rejections by @romange in #3387
• fix: remove fiber guard from non atomic section by @kostasrim in #3381
• chore: do not preempt on db_slice::RegisterOnChange by @kostasrim in #3388
• fix: disable inline transactions when db_slice has registered callbacks by @kostasrim in #3391
• fix: test_big_value_serialization_memory_limit shutdown timeout by @kostasrim in #3390
• chore: set serialization_max_chunk_size to 1 byte by @kostasrim in #3379
• chore: tiere...

v1.20.1

Dragonfly v1.20.1

This is a patch release.

fix: S3 access for loading/saving snapshots (#3296)

Signed-off-by: kostas kostas@dragonflydb.io

v1.20.0

Dragonfly v1.20.0

Some prominent changes include:

• Improvements around client side caching (#3136 #3158)
  Specifically, rueidis is now supported (#2454)
• Improvements around search and vector search APIs, FT.ALTER is now supported (#3144, #3148, #3178, #3186)
• JSON.MSET is added (#3167)
• Expiry notifications are now supported (#3154)

What's Changed

• test(cluster_mgr): Add tests for cluster_mgr.py by @chakaz in #3129
• chore: get rid of kv_args and replace it with slices to full_args by @romange in #3101
• fix: Fix live-lock in connection test by @chakaz in #3135
• chore: call breaker_cb_ on shutdown by @romange in #3128
• chore: Add option to disable aws dependency by @dranikpg in #3077
• feat(cluster): Add important logs by @chakaz in #3138
• chore: a small cleanup by @romange in #3140
• test: skip test_cluster_migration_cancel, it is broken by @BorysTheDev in #3146
• chore: update helio by @kostasrim in #3150
• fix: fix bug in cluster/slot_set by @BorysTheDev in #3143
• chore: recommit client tracking by @kostasrim in #3136
• feat(search): Tag field options (separator, case sensitivity) by @dranikpg in #3144
• feat(search): basic FT.ALTER by @dranikpg in #3148
• chore: Introduce pipeline back-pressure by @romange in #3152
• fix: lua and client tracking by @kostasrim in #3163
• feat: client tracking optout by @kostasrim in #3158
• feat: add noloop subcommand in client tracking by @kostasrim in #3164
• fix(ci): docker not using iouring by @kostasrim in #3169
• fix(ci): add missing docker option on reg tests workflow by @kostasrim in #3172
• fix: acl compatibility by @kostasrim in #3147
• chore: Streamer is rewritten with async interface by @romange in #3108
• feat(json): MSET by @dranikpg in #3167
• fix(cluster): Wait for flow creation before cancelling it by @chakaz in #3159
• chore: improve ft.create for vector search by @romange in #3178
• feat: add incoming migration error processing by @BorysTheDev in #3175
• chore: add parsing support for the rest of VECTOR parameters by @romange in #3179
• fix(cluster): Wait before all access to slot migrations by @chakaz in #3180
• fix(server): fix bug in replication on cached mode by @adiholden in #3156
• chore: small acl compat changes by @kostasrim in #3177
• chore: allow calling Context::ReportError from I/O dispatch fiber by @romange in #3182
• chore: add a lexer test covering KNN query by @romange in #3185
• chore(search): Add __vector_score alias by @dranikpg in #3186
• chore: Refactor string span management by @dranikpg in #3165
• chore(ci): run replication tests on arm by @kostasrim in #3168
• chore(search): improve parser error reporting by @romange in #3184
• fix(reply_builder): remove virtual modifier in SendError(ErrorReply) method by @BagritsevichStepan in #3191
• fix(cluster): Support FLUSHALL while slot migration is in progress by @chakaz in #3173
• fix: fix RegisterOnChange methods for journal and db_slice by @BorysTheDev in #3171
• fix(generic_family): fix RenameGeneric command for non-string data types by @BagritsevichStepan in #3181
• fix(server): Rename confusing flag replica_reconnect_on_master_restart by @chakaz in #3193
• fix(unit tests): fix generic family info test by @adiholden in #3187
• test: improve cluster_fuzzy_migration test by @BorysTheDev in #3197
• chore(core): Remove DfImpl inside ScoredMap by @dranikpg in #3199
• chore(tiering): Remove IoMgr by @dranikpg in #3198
• test(cluster): Make sure migration maintains TTL by @chakaz in #3188
• feat(cluster): Support STICK bit in slot migration by @chakaz in #3200
• chore: Re-enable previously flaky test by @chakaz in #3196
• feat(acl): add support of multiple passwords by @kostasrim in #3189
• chore(tiering): More advanced tiering tests by @dranikpg in #3201
• chore: add replica-priority flag by @kostasrim in #3204
• chore: fix tiering regtest test by @dranikpg in #3208
• feat(generic_family): Assemble RESTORE and STICK commands into one commmand during replication of the RENAME command by @BagritsevichStepan in #3209
• core(search): Add EF_RUNTIME parameter by @dranikpg in #3206
• fix(sanitizers): failing json test by @kostasrim in #3176
• feat(server): expiry notifications by @dranikpg in #3154
• fix: replicaof_reject_on_load flake by @kostasrim in #3203
• fix(cluster): Don't miss keys when migrating slots by @chakaz in #3218
• fix(tiering): Fix pending leak during immediate stash error by @dranikpg in #3219
• chore: provide basic logging to catch possible command errors by @romange in #3213
• chore: disable tiering test by @dranikpg in #3236
• fix(transaction): Don't transactionalize empty EVAL inside EXEC by @dranikpg in #3231
• fix: fix deadlock in DFLYCLUSTER CONFIG command and outgoing migration finaliztion by @BorysTheDev in #3239
• chore(tiered): minor fixes + expose buffer allocation type stats by @romange in #3234
• fix(pytest): timed ticker for simpler conditions by @dranikpg in #3242
• fix(server): Fix SCAN deadlock by @dranikpg in #3235
• chore: introduce back-pressure to tiered storage by @romange in #3243
• chore: improve dfly_bench by @romange in #3244
• chore(tiering): External alloc free page chaining by @dranikpg in #3237
• fix: total_stash_overflows statistic by @romange in #3248
• chore: update action versions by @romange in #3250
• chore(lua): Return which undeclared key was accessed by @chakaz in #3245
• chore: replace session wide fixtures with scope by @kostasrim in #3251
• chore: pull helio and adjust mimalloc for 32MiB segments by @kostasrim in #3174
• chore: add more logs around the duplicate value case in small_bins by @romange in #3254
• fix: fix move error during migration finalization by @BorysTheDev in #3253
• chore: upload all the logs from /tmp by @romange in #3263
• fix(tiering): Throttle snapshot load by @dranikpg in #3249
• chore: initiate grow preemptively by @romange in #3267
• chore(monitoring): add more dashboards + memcached by @romange in #3268
• fix(json_family): fix JSON.GET commmand for JSON legacy mode by @BagritsevichStepan in #3261
• chore: dfly_bench can send traffic to memcached by @romange in #3271
• chore: print effective QPS of the server. by @romange in #3274
• chore: debugging fixes by @romange in https...

v1.19.2

Dragonfly v1.19.2

This is a patch release.

What's Changed

chore(lua): Return which undeclared key was accessed (#3245)

Urgency to update

Low

v1.19.1

Dragonfly v1.19.1

This is a patch release.

What's Changed

fix: allow SCANs running within lua scripts #3232
fix: avoid crashing when calling empty EVAL inside MULTI transaction #3230

v1.19.0

Dragonfly v1.19.0

Some prominent changes include:

• SPOP and SRANDMEMBER return truly randomized results (#3022)
• Fix memory blowup in some cases during replication (#3084 #3103)
• Snapshotting works with data tiering (#3073)
• Fix incompatibility issue of RDB snapshot with Redis 6.x (#3121)
• Memory utilization improvements + better introspection of memory usage via /metrics

What's Changed

• chore: update versions and remove caching code by @romange in #3030
• chore: fix macos tiering by @dranikpg in #3032
• feat: support subrange expressions in jsonpathv2 by @romange in #3036
• fix: reduce delay when stop replica #3020 by @BorysTheDev in #3028
• fix(set): fix random in SRANDMEMBER and SPOP commands by @BagritsevichStepan in #3022
• chore: prevent updating gcc on macos by @romange in #3037
• chore: add half-range indices by @romange in #3041
• chore: update helio together with new mimalloc version by @romange in #3040
• fix: header for oom_errors_total by @romange in #3043
• chore: reset enable_direct_fd to avoid socket leakage in kernel by @romange in #3051
• chore: update deprecated actions by @romange in #3052
• chore: remove unneeded check-fails by @romange in #3054
• chore: Export replication memory stats by @romange in #3062
• !: Update grafana panel by @MaoMaoCake in #3064
• chore: add replication memory stats to the dashboard by @romange in #3065
• chore: bpop prints by @dranikpg in #3076
• fix(tiering): wait for IO before test teardown by @dranikpg in #3079
• chore: small replayer fixes by @dranikpg in #3081
• fix: increase lua stack limit to 8KB by @romange in #3080
• chore: minor fixes by @romange in #3082
• fix: fix cluster incorrect keys status by @BorysTheDev in #3083
• fix(acl): return -NOPERM instead of response error by @kostasrim in #3049
• feat(server): Expose serialization bytes via INFO and /metrics by @chakaz in #3087
• chore(streams): Some refactoring by @dranikpg in #3088
• feat(server): remove multi shard sync from replication by @adiholden in #3085
• fix(replication): fullsync phase write to sync on noop by @adiholden in #3084
• fix(tiering): Async delete for small bins by @dranikpg in #3068
• fix: prevent crashing if error happened during snapshot moving by @BorysTheDev in #3092
• fix: add timeout for DFLYMIGRATE ACK to prevent deadlock by @BorysTheDev in #3093
• chore: improve Migration() by @kostasrim in #3033
• feat(streams): Stream optimizations by @dranikpg in #3091
• feat(tiering): Defragmentation by @dranikpg in #3021
• fix: bit shift of kTagMask by @haodongnj in #3099
• feat(cluster_mgr): Allow attaching replicas by @chakaz in #3105
• bug(server): fix replication stuck in full sync by @adiholden in #3103
• fix(server): Sync FLUSH with tiering by @dranikpg in #3098
• chore: export pipeline related metrics by @romange in #3104
• chore: zset error log (from 1.18 branch) by @dranikpg in #3109
• New test for cluster migration: connection issue by @BorysTheDev in #3102
• chore: minor fixes + remove redundant DCHECK by @romange in #3115
• fix: fix cluster_fuzzy_migration test by @BorysTheDev in #3116
• feat(server): Pipeline and dispatch on /metrics by @chakaz in #3117
• chore: pull helio add test for tls deadlock by @kostasrim in #3111
• feat(cluster_mgr): Improvements to cluster_mgr.py by @chakaz in #3118
• refactor client tracking, fix atomicity, squashing and multi/exec by @kostasrim in #2970
• fix(server): fix compatibility with rdb snapshot by @adiholden in #3121
• Revert "refactor client tracking, fix atomicity, squashing and multi/… by @adiholden in #3122
• feat(cluster_mgr): Take over command by @chakaz in #3120
• fix: fix RestoreStreamer to prevent buckets skipping #2830 by @BorysTheDev in #3119
• feat(cluster_mgr): Fix migration action by @chakaz in #3124
• chore: prevent Dispatch fiber to be launched during migration by @romange in #3123
• feat(tiering): Simple snapshotting by @dranikpg in #3073
• fix: upload action in version release by @romange in #3127

New Contributors

• @MaoMaoCake made their first contribution in #3064
• @haodongnj made their first contribution in #3099

Huge thanks to all the contributors! ❤️

Full Changelog: v1.18.0...v1.19.0

v1.18.1

Dragonfly v1.18.1

This is a patch release.

What's Changed

fix: prevent socket leakage in kernel with io_uring mode (#3051)