v1.19.2

Dragonfly v1.19.2

This is a patch release.

What's Changed

chore(lua): Return which undeclared key was accessed (#3245)

Urgency to update

Low

v1.19.1

Dragonfly v1.19.1

This is a patch release.

What's Changed

fix: allow SCANs running within lua scripts #3232
fix: avoid crashing when calling empty EVAL inside MULTI transaction #3230

v1.19.0

Dragonfly v1.19.0

Some prominent changes include:

• SPOP and SRANDMEMBER return truly randomized results (#3022)
• Fix memory blowup in some cases during replication (#3084 #3103)
• Snapshotting works with data tiering (#3073)
• Fix incompatibility issue of RDB snapshot with Redis 6.x (#3121)
• Memory utilization improvements + better introspection of memory usage via /metrics

What's Changed

• chore: update versions and remove caching code by @romange in #3030
• chore: fix macos tiering by @dranikpg in #3032
• feat: support subrange expressions in jsonpathv2 by @romange in #3036
• fix: reduce delay when stop replica #3020 by @BorysTheDev in #3028
• fix(set): fix random in SRANDMEMBER and SPOP commands by @BagritsevichStepan in #3022
• chore: prevent updating gcc on macos by @romange in #3037
• chore: add half-range indices by @romange in #3041
• chore: update helio together with new mimalloc version by @romange in #3040
• fix: header for oom_errors_total by @romange in #3043
• chore: reset enable_direct_fd to avoid socket leakage in kernel by @romange in #3051
• chore: update deprecated actions by @romange in #3052
• chore: remove unneeded check-fails by @romange in #3054
• chore: Export replication memory stats by @romange in #3062
• !: Update grafana panel by @MaoMaoCake in #3064
• chore: add replication memory stats to the dashboard by @romange in #3065
• chore: bpop prints by @dranikpg in #3076
• fix(tiering): wait for IO before test teardown by @dranikpg in #3079
• chore: small replayer fixes by @dranikpg in #3081
• fix: increase lua stack limit to 8KB by @romange in #3080
• chore: minor fixes by @romange in #3082
• fix: fix cluster incorrect keys status by @BorysTheDev in #3083
• fix(acl): return -NOPERM instead of response error by @kostasrim in #3049
• feat(server): Expose serialization bytes via INFO and /metrics by @chakaz in #3087
• chore(streams): Some refactoring by @dranikpg in #3088
• feat(server): remove multi shard sync from replication by @adiholden in #3085
• fix(replication): fullsync phase write to sync on noop by @adiholden in #3084
• fix(tiering): Async delete for small bins by @dranikpg in #3068
• fix: prevent crashing if error happened during snapshot moving by @BorysTheDev in #3092
• fix: add timeout for DFLYMIGRATE ACK to prevent deadlock by @BorysTheDev in #3093
• chore: improve Migration() by @kostasrim in #3033
• feat(streams): Stream optimizations by @dranikpg in #3091
• feat(tiering): Defragmentation by @dranikpg in #3021
• fix: bit shift of kTagMask by @haodongnj in #3099
• feat(cluster_mgr): Allow attaching replicas by @chakaz in #3105
• bug(server): fix replication stuck in full sync by @adiholden in #3103
• fix(server): Sync FLUSH with tiering by @dranikpg in #3098
• chore: export pipeline related metrics by @romange in #3104
• chore: zset error log (from 1.18 branch) by @dranikpg in #3109
• New test for cluster migration: connection issue by @BorysTheDev in #3102
• chore: minor fixes + remove redundant DCHECK by @romange in #3115
• fix: fix cluster_fuzzy_migration test by @BorysTheDev in #3116
• feat(server): Pipeline and dispatch on /metrics by @chakaz in #3117
• chore: pull helio add test for tls deadlock by @kostasrim in #3111
• feat(cluster_mgr): Improvements to cluster_mgr.py by @chakaz in #3118
• refactor client tracking, fix atomicity, squashing and multi/exec by @kostasrim in #2970
• fix(server): fix compatibility with rdb snapshot by @adiholden in #3121
• Revert "refactor client tracking, fix atomicity, squashing and multi/… by @adiholden in #3122
• feat(cluster_mgr): Take over command by @chakaz in #3120
• fix: fix RestoreStreamer to prevent buckets skipping #2830 by @BorysTheDev in #3119
• feat(cluster_mgr): Fix migration action by @chakaz in #3124
• chore: prevent Dispatch fiber to be launched during migration by @romange in #3123
• feat(tiering): Simple snapshotting by @dranikpg in #3073
• fix: upload action in version release by @romange in #3127

New Contributors

• @MaoMaoCake made their first contribution in #3064
• @haodongnj made their first contribution in #3099

Huge thanks to all the contributors! ❤️

Full Changelog: v1.18.0...v1.19.0

v1.18.1

Dragonfly v1.18.1

This is a patch release.

What's Changed

fix: prevent socket leakage in kernel with io_uring mode (#3051)

v1.18.0

Dragonfly v1.18.0

Some prominent changes include:

• ACL improvements: #2945 #2943 #2920 #2982 #2995
• Implementation of json.merge #2960
• Replication - memory improvements
• Very much alpha support for data tiering. Try it out with --tiered_prefix=/pathto/ssd/base and see how your memory usage goes down (STRING type only). Do not use it in prod! 😸

What's Changed

• feat: retry ACK if the configs are different #2833 by @BorysTheDev in #2906
• chore(tiering): Update Get, Set, Del by @dranikpg in #2897
• chore: preparation step for lock fingerprints by @romange in #2899
• fix(transaction): Use FinishHop in schedule by @dranikpg in #2911
• chore(tiering): Fix MacOs build by @dranikpg in #2913
• feat(cluster): Migration cancellation support by @chakaz in #2869
• feat: process migration data after FIN opcode #2864 by @BorysTheDev in #2918
• chore(string_family): Refactor SetCmd by @dranikpg in #2919
• fix: Improve reply latency of HELLO by @romange in #2925
• chore: improve reply latency of SendScoredArray by @romange in #2929
• Namespace support in prometheus rule by @Pothulapati in #2931
• fix: socket closed when RegisterOnErrorCb is called in HandleRequests by @kostasrim in #2932
• chore: bring more clarity when replayer fails by @romange in #2933
• Slot migration cancel crash fix by @BorysTheDev in #2934
• feat: add ability reaply config with migration #2924 by @BorysTheDev in #2926
• fix(test): Unflake fuzzy cluster migration test by @chakaz in #2927
• chore: Remove Schedule() call by @romange in #2938
• chore: get rid of lock keys by @romange in #2894
• fix: introduce info_replication_valkey_compatible flag by @romange in #2936
• feat(metrics): adding max_clients to metrics and info output (#2912) by @racamirko in #2940
• chore: adjust transaction code to keystep/3 commands by @romange in #2941
• feat(tiering): Get, GetSet, Set test by @dranikpg in #2921
• chore(acl): adjust some ACL command responses by @Niennienzz in #2943
• chore: Pull helio with new future by @dranikpg in #2944
• refactor: add cluster namespace by @BorysTheDev in #2948
• chore: Introduce ShardArgs as a distinct type by @romange in #2952
• chore: Log db_index in traffic logger by @dranikpg in #2951
• fixes for v1.18.0 by @adiholden in #2956
• feat(tiering): Support append (and modifications in general) by @dranikpg in #2949
• feat: extended bracket index in jsonpath by @romange in #2954
• chore: Remove TieringV1 by @dranikpg in #2962
• fix(pytests): replace proc.wait() with proc.communicate() to avoid deadlocks by @kostasrim in #2964
• feat(tiering): Registered buffers by @dranikpg in #2967
• feat: add slot migration error processing by @BorysTheDev in #2957
• chore(acl): allow multiple users in acl deluser by @kostasrim in #2945
• feat: implement json.merge by @romange in #2960
• fix: fix deadlock and slot flush for migration cancel #2968 by @BorysTheDev in #2972
• chore(tiering): Lots of metrics by @dranikpg in #2977
• fix: crash during migration when connection is closing by @kostasrim in #2973
• fix: remove acl-check and cancel instead when REPLCONF ACK fails to validate by @kostasrim in #2920
• fix: check return code of process after communicate by @kostasrim in #2976
• fix: allow non hashed passwords when loading users from acl file by @kostasrim in #2982
• chore: update our container distributions versions by @romange in #2983
• chore: remove version checks when running our regtests by @romange in #2988
• chore(acl): add vlog and check on deluser flow by @kostasrim in #2995
• fix(memcached): Register memcached listener to handle --maxclients by @chakaz in #2985
• chore: another preparation commit to get rid of kv_args in transaction by @romange in #2996
• chore: improve performance of Scan operation by @romange in #2990
• fix(server): small string allocations only under 256 bytes str by @adiholden in #2991
• fix(cluster-migration): Support cancelling migration right after starting it by @chakaz in #2992
• chore: fix double header issue by @romange in #3002
• chore: small tiering fixes by @dranikpg in #2966
• feat(benchmark-tests): run in K8s by @zacharya19 in #2965
• Benchmark fixes by @zacharya19 in #3005
• fix(tiering): rename v2 + max_file_size by @dranikpg in #3004
• chore: fix tiering macos stub by @dranikpg in #3006
• chore: export listener stats by @romange in #3007
• chore: pull latest helio by @romange in #3009
• fix(server): lag is 0 when server not in stable state by @adiholden in #3010
• chore: get rid of kv_args and replace it with slices to full_args by @romange in #2942
• fix(server): non auto journal write after callback finish by @adiholden in #3016
• fix(server): shrink replication steaming buf by @adiholden in #3012
• fix(zset): fix random in ZRANDMEMBER command by @BagritsevichStepan in #2994
• Fix benchmark by @adiholden in #3017
• chore: Remove tiering test skip by @dranikpg in #3011
• feat(tiering): simple offload loop by @dranikpg in #2987
• feat(tiering): MGET support by @dranikpg in #3013
• Revert "chore: get rid of kv_args and replace it with slices to full_… by @romange in #3024
• chore(dash): Replace comparator with predicate by @dranikpg in #3025
• feat: add defragment command by @BorysTheDev in #3003

Huge thanks to all the contributors! ❤️

New Contributors

• @racamirko made their first contribution in #2940
• @BagritsevichStepan made their first contribution in #2994

Full Changelog: v1.17.0...v1.18.0

v1.17.1

Dragonfly v1.17.1

This is a patch release.

What's Changed

fix: potential data race in the transactional code (#2911)

v1.17.0

Dragonfly v1.17.0

Some prominent changes include:

• Improved performance for MGET operations (#2453)
• Fix argument parsing in json.objkeys (#2872)
• Fix ipv6 support for replication (#2889)
• Support serialisation of bloom filters - saving to and loading from snapshots (#2846)
• Support of HLL PFADD (#2761)
• Support bullmq workloads that do not have {} hashtags in their queue names (#2890)

What's Changed

• fix: #2745 don't start migration process again after apply the same the same config is applied by @BorysTheDev in #2822
• feat(transaction): Idempotent callbacks (immediate runs) by @dranikpg in #2453
• refactor(cluster): replace sync_id with node_id for slot migration #2835 by @BorysTheDev in #2838
• feat(tiering): Simple OpManager by @dranikpg in #2781
• chore: implement path mutation for JsonFlat by @romange in #2805
• feat(cluster): add migration removing by config #2835 by @BorysTheDev in #2844
• chore: expose direct API on Bloom objects by @romange in #2845
• chore: generalize CompactObject::AllocateMR by @kostasrim in #2847
• feat(tiering): Simplest small bins by @dranikpg in #2810
• refactor: clean cluster slot migration code by @BorysTheDev in #2848
• fix(tests): Fix numsub test by @dranikpg in #2852
• fix: healthcheck for docker containers by @romange in #2853
• fix: possible crash in tls code by @romange in #2854
• fix(server): Do not block admin-port commands by @chakaz in #2842
• fix(pytest): make pytests fail if server crash on shutdown by @adiholden in #2827
• feat(server): add prints on takeover timeout by @adiholden in #2856
• fix(pytest): dont check process return code on kill by @adiholden in #2862
• fix: authorize the http connection to call commands by @romange in #2863
• feat(cluster): Send number of keys for incoming and outgoing migrations. by @chakaz in #2858
• feat(tiering): TieredStorageV2 by @dranikpg in #2849
• bug(server): set connection flags block/pause flag on all blocking commands by @adiholden in #2816
• chore: serialize SBF by @romange in #2846
• fix: test_replicaof_reject_on_load crash on stop by @kostasrim in #2818
• feat(dbslice): Add self-laundering iterator in DbSlice by @chakaz in #2815
• chore: License update by @romange in #2767
• fix(acl): incompatibilities with acl load by @kostasrim in #2867
• fix(json): make path optional in json.objkeys by @kostasrim in #2872
• fix: return wrong type errors for SET...GET command by @romange in #2874
• fix(redis replication): remove partial sync flow ,not supported yet by @adiholden in #2865
• chore: limit traffic logger only to the main interface by @romange in #2877
• chore: relax repltakeover constraints to only exclude write commands by @kostasrim in #2873
• chore(replayer): Roll back to go1.18 by @dranikpg in #2881
• fix: brpoplpush single shard to wake up blocked transactions by @kostasrim in #2875
• chore: LockTable tracks fingerprints of keys by @romange in #2839
• chore: reject TLS handshake when our listener is plain TCP by @romange in #2882
• Add support for Sparse HLL PFADD by @azuredream in #2761
• feat server: bring visibility to script errors by @adiholden in #2879
• chore: clean up REPLTAKEOVER flow by @romange in #2887
• chore(tiering): Move files and move kb literal to common by @dranikpg in #2868
• chore(interpreter): Support object replies by @dranikpg in #2885
• fix(ci/helm): Stick to v0.73.0 version of prom operator by @Pothulapati in #2893
• fix(acl): authentication with UDS socket by @kostasrim in #2895
• feat(cluster): add repeated ACK if an error is happened by @BorysTheDev in #2892
• chore(blocking): Remove faulty DCHECK by @dranikpg in #2898
• chore: add a clear link on how to build dragonfly from source by @romange in #2884
• feat(server): Allow configuration of hashtag extraction by @chakaz in #2890
• fix: fix build under macos by @BorysTheDev in #2901
• fix(cluster_replication): replicate redis cluster node bug fix by @adiholden in #2876
• fix(acl): skip http and add check on connection traversals by @kostasrim in #2883
• fix(zset): Better memory consumption calculation by @chakaz in #2900
• fix: fix ld for num converting by @BorysTheDev in #2902
• chore: add help string for memory_fiberstack_vms_bytes by @romange in #2903
• fix(sanitizers): false positive fail on multi_test::Eval by @kostasrim in #2896
• chore: pull helio and add ipv6 replication test by @dranikpg in #2889
• chore: add ipv6 support for native linux release by @romange in #2908

Huge thanks to all the contributors! ❤️

Full Changelog: v1.16.0...v1.17.0

v1.16.1

Dragonfly v1.16.1

This is a patch release.

What's Changed

• fix: authorize the http connection to call DF commands (#2863)
• possible crash in tls code (#2854 )
• healthcheck for docker containers (#2853)

Full Changelog: v1.16.0...v1.16.1

v1.16.0

Dragonfly v1.16.0

Our spring release. We are getting closer to 2.0 with some very exciting features ahead. Stay tuned!

Some prominent changes include:

• Improved memory accounting of client connections (#2710 #2755 and #2692 )
• FT.AGGREGATE call (#2413)
• Properly handle and replicate Memcache flags (#2787 #2807)
• Intoduce BF.AGGREGATE BD.(M)ADD and BF.(M)EXISTS methods (#2801). Note, that it does not work with snapshots and replication yet.
• Dragonfly builds natively on MacOS. We would love some help with extending the release pipeline to create a proper macos binary.
• Following the requests from the Edge developers community, we added a basic HTTP API support! Try running Dragonfly with:
  --expose_http_api flag and then call curl -X POST -d '["ping"]' localhost:6379/api. We will follow up with more extensive docs later this month.
• Lots of stability fixes, especially around Sidekiq and BullMQ workloads.

What's Changed

• chore: make usan asan optional and enable them on CI by @kostasrim in #2631
• fix: missing manual trigger for daily sanitizers by @kostasrim in #2682
• bug(tiering): fix overflow in page offset calculation and wrong hash offset calculation by @theyueli in #2683
• Chore: Fixed Docker Health Check by @manojks1999 in #2659
• chore: Increase disk space in the coverage runs by @romange in #2684
• fix(flushall): Decommit memory after releasing tables. by @chakaz in #2691
• feat(server): Account for serializer's temporary buffer size by @chakaz in #2689
• refactor: remove FULL-SYNC-CUT cmd #2687 by @BorysTheDev in #2688
• chore: add malloc-based stats and decommit by @romange in #2692
• feat(cluster): automatic slot migration finalization #2697 by @BorysTheDev in #2698
• Basic FT.AGGREGATE by @dranikpg in #2413
• chore: little transaction cleanup by @dranikpg in #2608
• fix(channel store): add acquire/release pair in fast update path by @dranikpg in #2704
• chore: add ubuntu22 devcontainer by @romange in #2700
• feat(cluster): Add --cluster_id flag by @chakaz in #2695
• feat(server): Use mimalloc in SSL calls by @chakaz in #2710
• chore: Pull helio with new BlockingCounter by @dranikpg in #2711
• chore(transaction): Simplify PollExecution by @dranikpg in #2712
• chore(transaction): Don't call GetLocalMask from blocking controller by @dranikpg in #2715
• chore: improve compatibility of EXPIRE functions with Redis by @romange in #2696
• chore: disable flaky fuzzy migration test by @kostasrim in #2716
• chore: update sanitizers workflow by @kostasrim in #2686
• chore: Use c-ares for resolving hosts in ProtocolClient by @chakaz in #2719
• Remove check-fail in ExpireIfNeeded and introduce DFLY LOAD by @romange in #2699
• chore: Record cmd stat from invoke by @dranikpg in #2720
• fix(transaction): nullptr access on non transactional commands by @kostasrim in #2724
• fix(BgSave): async from sync by @kostasrim in #2702
• chore: remove core/fibers by @romange in #2723
• fix(transaction): Replace with armed sync point by @dranikpg in #2708
• feat(json): Deserialize ReJSON format by @chakaz in #2725
• feat: add flag masteruser by @kostasrim in #2693
• refactor: block list refactoring #2580 by @BorysTheDev in #2732
• chore: fix DeduceExecMode by @dranikpg in #2733
• fix(cluster): Reply with correct \n / \r\n from CLUSTER sub cmd by @chakaz in #2731
• chore: Introduce fiber stack allocator by @romange in #2730
• fix(cluster): Save replica ID per replica by @chakaz in #2735
• fix(ssl): Proper cleanup by @chakaz in #2742
• chore: add skeleton files for flat_dfs code by @romange in #2738
• chore: better error reporting when connecting to tls with plain socket by @romange in #2740
• chore: Support json paths without root selector by @dranikpg in #2747
• chore: journal cleanup by @romange in #2749
• refactor: remove start-slot-migration cmd #2727 by @BorysTheDev in #2728
• feat(server): Add TLS usage to /metrics and INFO MEMORY by @chakaz in #2755
• chore: preparations for adding flat json support by @romange in #2752
• chore(transaction): Introduce RunCallback by @dranikpg in #2760
• feat(replication): Do not auto replicate different master by @chakaz in #2753
• Improve Helm chart to be rendered locally and on machines where is not the application target by @fafg in #2706
• chore: preparation for basic http api by @romange in #2764
• feat(server): Add metric for RDB save duration. by @chakaz in #2768
• chore: fix flat_dfs read tests. by @romange in #2772
• fix(ci): do not overwrite last_log_file among tests by @kostasrim in #2759
• feat(server): support cluster replication by @adiholden in #2748
• fix: fiber preempts on read path and OnCbFinish() clears fetched_items_ by @kostasrim in #2763
• chore(ci): open last_log_file in append mode by @kostasrim in #2776
• doc(README): fix outdated expiry ranges description by @enjoy-binbin in #2779
• Benchmark runner by @adiholden in #2780
• chore(replication-tests): add cache_mode on test replication all by @kostasrim in #2685
• feat(tiering): DiskStorage by @dranikpg in #2770
• chore: add a boilerplate for bloom filter family by @romange in #2782
• chore: introduce conversion routines between JsonType and FlatJson by @romange in #2785
• chore: Fix memcached flags not updated by @dranikpg in #2787
• chore: remove duplicate code from dash and simplify by @kostasrim in #2765
• chore: disable test_cluster_slot_migration by @kostasrim in #2788
• fix: new[] delete[] missmatch in disk_storage by @kostasrim in #2792
• fix: sanitizers clang build and clean up some warnings by @kostasrim in #2793
• chore: add bloom filter class by @romange in #2791
• chore: add SBF data structure by @romange in #2795
• chore(tiering): Disable compilation for MacOs by @dranikpg in #2799
• chore: fix daily build by @romange in #2798
• chore: expose SBF via compact_object by @romange in #2797
• fix(ci): malloc trim on sanitizers workflow by @kostasrim in #2794
• fix(cluster): Don't miss updates in FLUSHSLOTS by @chakaz in #2783
• feat: add bf.(m)add and bf.(m)exists commands by @romange in #2801
• fix: SBF memory leaks by @kostasrim in #2803
• chore: refactor StringFamily::Set to use CmdArgParser by @romange in #2800
• fix: propagate memcached flags to replica by @romange in #2807
• DFLYMIGRATE ACK refactoring by @BorysTheDev in #2790
• feat: add master lsn and journal_executed dcheck in replica via ping by @kostasrim in #277...

v1.15.1

Dragonfly v1.15.1

This is a patch release.

What's Changed

fix: Workaround for a possible crash during BLPOP calls.

Full Changelog: v1.15.0...v1.15.1