Skip to content

fix: prevent crash on KEYLOCK_ACQUIRED check for NO_KEY_TRANSACTIONAL commands #5185

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 10 commits into from
May 29, 2025
64 changes: 64 additions & 0 deletions src/server/search/search_family_test.cc
Original file line number Diff line number Diff line change
Expand Up @@ -2768,6 +2768,69 @@ TEST_F(SearchFamilyTest, JsonSetIndexesBug) {
EXPECT_THAT(resp, IsUnordArrayWithSize(IsMap("text", "some text")));
}

TEST_F(SearchFamilyTest, SearchReindexWriteSearchRace) {
const std::string kIndexName = "myRaceIdx";
const int kWriterOps = 2000; // Number of documents to write
const int kSearcherOps = 1500; // Number of search operations
const int kReindexerOps = 25; // Number of times to drop/recreate the index

// 1. Initial FT.CREATE for the index
// Schema from the issue: content TEXT SORTABLE, tags TAG SORTABLE, numeric_field NUMERIC SORTABLE
Run({"ft.create", kIndexName, "ON", "HASH", "PREFIX", "1", "doc:", "SCHEMA", "content", "TEXT",
"SORTABLE", "tags", "TAG", "SORTABLE", "numeric_field", "NUMERIC", "SORTABLE"});

// 2. writer_fiber
auto writer_fiber = pp_->at(0)->LaunchFiber([&] {
for (int i = 1; i <= kWriterOps; ++i) {
std::string doc_key = absl::StrCat("doc:", i);
std::string content = absl::StrCat("text data item ", i, " for race condition test");
std::string tags_val = absl::StrCat("tagA,tagB,", (i % 10));
std::string numeric_field_val = std::to_string(i);
Run({"hset", doc_key, "content", content, "tags", tags_val, "numeric_field",
numeric_field_val});
if (i % 100 == 0)
ThisFiber::SleepFor(std::chrono::microseconds(100)); // Brief yield
}
});

// 3. searcher_fiber
auto searcher_fiber = pp_->at(1)->LaunchFiber([&] {
for (int i = 1; i <= kSearcherOps; ++i) {
int random_val_content = 1 + (i % kWriterOps);
int random_tag_val = i % 10;
int random_val_numeric = 1 + (i % kWriterOps);

std::string query_content = absl::StrCat("@content:item", random_val_content);
std::string query_tags = absl::StrCat("@tags:{tagA|tagB|tag", random_tag_val, "}");
std::string query_numeric = absl::StrCat("@numeric_field:[", random_val_numeric, " ",
(random_val_numeric + 100), "]");
Run({"ft.search", kIndexName, query_content});
Run({"ft.search", kIndexName, query_tags});
Copy link
Collaborator

@romange romange May 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am pretty sure the bug is unrelated to query contents.
the goal is to provide a minimal reproducible example - everything else is just noise.

  1. please make the query constant
  2. there is no need for 3 calls. you can increase kSearcherOps if needed

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I made the test smaller

Run({"ft.search", kIndexName, query_numeric});
if (i % 50 == 0)
ThisFiber::SleepFor(std::chrono::microseconds(200 * (1 + i % 2)));
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suspect that a constant will suffice here

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I removed all sleeps. The bug is reproducible without them.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

even better.

}
});

// 4. reindexer_fiber
auto reindexer_fiber = pp_->at(2)->LaunchFiber([&] {
for (int i = 1; i <= kReindexerOps; ++i) {
Run({"ft.create", kIndexName, "ON", "HASH", "PREFIX", "1", "doc:", "SCHEMA", "content",
"TEXT", "SORTABLE", "tags", "TAG", "SORTABLE", "numeric_field", "NUMERIC", "SORTABLE"});
ThisFiber::SleepFor(std::chrono::milliseconds(10 + (i % 5 * 5)));
Run({"ft.dropindex", kIndexName});
ThisFiber::SleepFor(std::chrono::microseconds(500 * (1 + i % 2)));
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are you sure you need reindexer_fiber at all?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried to remove every part. This test is as minimal as I could create to reproduce.

}
});

// Join fibers
writer_fiber.Join();
searcher_fiber.Join();
reindexer_fiber.Join();

ASSERT_FALSE(service_->IsShardSetLocked());
}

TEST_F(SearchFamilyTest, IgnoredOptionsInFtCreate) {
Run({"HSET", "doc:1", "title", "Test Document"});

Expand Down Expand Up @@ -2801,4 +2864,5 @@ TEST_F(SearchFamilyTest, IgnoredOptionsInFtCreate) {
resp = Run({"FT.SEARCH", "idx", "*"});
EXPECT_THAT(resp, AreDocIds("doc:1"));
}

} // namespace dfly
13 changes: 10 additions & 3 deletions src/server/transaction.cc
Original file line number Diff line number Diff line change
Expand Up @@ -1245,9 +1245,16 @@ bool Transaction::CancelShardCb(EngineShard* shard) {
shard->shard_lock()->Release(LockMode());
} else {
auto lock_args = GetLockArgs(shard->shard_id());
DCHECK(sd.local_mask & KEYLOCK_ACQUIRED);
DCHECK(!lock_args.fps.empty());
GetDbSlice(shard->shard_id()).Release(LockMode(), lock_args);

if (lock_args.fps.empty()) {
// For NO_KEY_TRANSACTIONAL commands, we don't need to release locks
// because we don't acquire locks for them.
DCHECK(cid_->opt_mask() & CO::NO_KEY_TRANSACTIONAL);
} else {
DCHECK(sd.local_mask & KEYLOCK_ACQUIRED);
GetDbSlice(shard->shard_id()).Release(LockMode(), lock_args);
}

sd.local_mask &= ~KEYLOCK_ACQUIRED;
}

Expand Down
Loading