Valkey Fuzzer

[uriyage]

Add Fuzzing Capability to Valkey

Overview

This PR adds a fuzzing capability to Valkey, allowing developers and users to stress test their Valkey deployments with randomly generated commands. The fuzzer is integrated with the existing valkey-benchmark tool, making it easy to use without requiring additional dependencies.

Key Features

• Command Generator: Automatically generates Valkey commands by retrieving command information directly from the server
• Two Fuzzing Modes:

• normal: Generates only valid commands, doesn't modify server configurations
• aggressive: Includes malformed commands and allows CONFIG SET operations

• Multi-threaded Testing: Each client runs in a dedicated thread to maximize interaction between clients and enable testing of complicated scenarios

• Integration with valkey-benchmark: Uses the existing CLI interface

Implementation Details

• Added new files:

• fuzzer_command_generator.h/c: Dynamically generates valkey commands.
• fuzzer_client.c: Orchestrate all the client threads, report test progress, and handle errors.

• Modified existing files:

• valkey-benchmark.c: Added fuzzing mode options and integration

Command Generation Approach

The fuzzer dynamically retrieves command information from the server, allowing it to adapt to different Valkey versions and custom modules. Since the command information generated from JSON files is sometimes limited, not all generated commands will be valid, but approximately 95% valid command generation is achieved.

It is important to generate valid commands to cover as much code path as possible and not just the invalid command/args path. The fuzzer prioritizes generating syntactically and semantically correct commands to ensure thorough testing of the server's core functionality, while still including a small percentage of invalid commands in aggressive mode to test error handling paths

Config modification

For CONFIG SET command, the situation is more complex as the server currently provides limited information through CONFIG GET *. Some hardcoded logic is implemented that will need to be modified in the future. Ideally, the server should provide self-inspection commands to retrieve config keys-values with their properties (enum values, modifiability status, etc.).

Issue Detection

The fuzzer is designed to identify several types of issues:
• Server crashes
• Server memory corruptions / memory leaks(when compiled with ASAN)
• Server unresponsiveness
• Server malformed replies

For unresponsiveness detection, command timeout limits are implemented to ensure no command blocks for excessive periods. If a server doesn't respond within 30 seconds, the fuzzer signals that something is wrong.

Proven Effectiveness

When running against the latest unstable version, the fuzzer has already identified several issues, demonstrating its effectiveness:

• [CRASH] crash when unblocking a client on timeout #2111
• [BUG] Use after free with lua-debug #2112
• Only mark the client reprocessing flag when unblocked on keys #2109
• fix commandlog argument schema to use pure tokens #2113
• Fix a case of out of bound read when cluster node ID is provided with wrong length in CLUSTER FORGET #2108
• Fix memory corruption in sharded pubsub unsubscribe #2137
• [BUG] missing response when AUTH is errored inside a transaction #2106
• [CRASH] Fix missing check for executing client #2347

How to Use

Run the fuzzer using the valkey-benchmark tool with the --fuzz flag:

# Basic usage (10000 commands 1000 commands per client, 10 clients)
./src/valkey-benchmark --fuzz -h 127.0.0.1 -p 6379 -n 10000 -c 10

# With aggressive fuzzing mode
./src/valkey-benchmark --fuzz --fuzz-level aggressive -h 127.0.0.1 -p 6379 -n 10000 -c 10

# With detailed logging
./src/valkey-benchmark --fuzz --fuzz-log-level debug -h 127.0.0.1 -p 6379 -n 10000 -c 10

The fuzzer supports existing valkey-benchmark options, including TLS and cluster mode configuration.

[codecov]

Codecov Report

Attention: Patch coverage is 0.54682% with 1455 lines in your changes missing coverage. Please review.

Please upload report for BASE (unstable@30ea139). Learn more about missing BASE report.
Report is 5 commits behind head on unstable.

Files with missing lines	Patch %	Lines
src/fuzzer_command_generator.c	0.00%	1021 Missing ⚠️
src/fuzzer_client.c	0.00%	421 Missing ⚠️
src/valkey-benchmark.c	38.09%	13 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##             unstable    #2340   +/-   ##
===========================================
  Coverage            ?   69.94%           
===========================================
  Files               ?      125           
  Lines               ?    68556           
  Branches            ?        0           
===========================================
  Hits                ?    47953           
  Misses              ?    20603           
  Partials            ?        0           

Files with missing lines	Coverage Δ
src/valkey-benchmark.c	61.12% <38.09%> (ø)
src/fuzzer_client.c	0.00% <0.00%> (ø)
src/fuzzer_command_generator.c	0.00% <0.00%> (ø)

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[hpatro]

Nice work @uriyage! Gave it a run locally, works pretty well.

I had a question about the final report, do you manually analyze all the errors and spot the issues, what is your mechanism around it?

[ranshid]

@uriyage shell we also add a daily job to run the fuzzer?

[hpatro]

@uriyage shell we also add a daily job to run the fuzzer?

I also thought about this but unsure about the triage part?

[uriyage]

Nice work @uriyage! Gave it a run locally, works pretty well.

I had a question about the final report, do you manually analyze all the errors and spot the issues, what is your mechanism around it?

The errors can be divided into two groups: server-side-observable and client-side-observable.

For server-side issues, we can identify crashes, memory corruptions, or memory leaks. With client-side issues,
we can observe timeouts (where the server is unexpectedly unresponsive) or malformed replies from the server
that indicate problems with reply generation.

For the first two server-side issues, which are the most common, we can simply validate that after the fuzzer run the server didn't crash and validate that no memory issues were reported (when compiled with ASAN).

Client-side issues are more difficult to root cause and require manual work to understand what went wrong. We
could potentially add in the future more server-side debug capabilities, such as having the server inspect its own output and
crash whenever it sends a malformed reply. We could also add some thread monitoring capability to identify when the server
becomes unresponsive.

[uriyage]

@uriyage shell we also add a daily job to run the fuzzer?

Thanks, I added it to the TCL tests so it will run with all the variations we currently use for TCL testing. Currently, the test will be considered a failure only if the server crashes or becomes unresponsive after the fuzzer run