[ENH] Add where collection filtering by metadata to distributed

[jairad26]

Description of changes

Summarize the changes made by this PR.

• Improvements & Bug fixes
  • ...
• New functionality
  • ...

Test plan

How are these changes tested?

• Tests pass locally with pytest for python, yarn test for js, cargo test for rust

Documentation Changes

Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the docs section?

[github-actions]

Reviewer Checklist

Please leverage this checklist to ensure your code review is thorough before approving

Testing, Bugs, Errors, Logs, Documentation

• Can you think of any use case in which the code does not behave as intended? Have they been tested?
• Can you think of any inputs or external events that could break the code? Is user input validated and safe? Have they been tested?
• If appropriate, are there adequate property based tests?
• If appropriate, are there adequate unit tests?
• Should any logging, debugging, tracing information be added or removed?
• Are error messages user-friendly?
• Have all documentation changes needed been made?
• Have all non-obvious changes been commented?

System Compatibility

• Are there any potential impacts on other parts of the system or backward compatibility?
• Does this change intersect with any items on our roadmap, and if so, is there a plan for fitting them together?

Quality

• Is this code of a unexpectedly high quality (Readability, Modularity, Intuitiveness)

[jairad26]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• [ENH] Add where collection filtering by metadata to distributed #5029 👈 (View in Graphite)
• [ENH] Add where filtering support on list collections for local chroma #5019
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[blacksmith-sh]

4 Jobs Failed:

PR checks / Python tests / test-rust-bindings (3.9, chromadb/test/property/test_cross_version_persist.py)

Step "Test" from job "Python tests / test-rust-bindings (3.9, chromadb/test/property/test_cross_version_persist.py)" is failing. The last 20 log lines are:

[...]
22.15s setup    chromadb/test/property/test_cross_version_persist.py::test_cycle_versions[version_settings2]
21.81s setup    chromadb/test/property/test_cross_version_persist.py::test_cycle_versions[version_settings1]
0.61s teardown chromadb/test/property/test_cross_version_persist.py::test_cycle_versions[version_settings1]
0.59s teardown chromadb/test/property/test_cross_version_persist.py::test_cycle_versions[version_settings0]
0.25s teardown chromadb/test/property/test_cross_version_persist.py::test_cycle_versions[version_settings2]
=========================== short test summary info ============================
FAILED chromadb/test/property/test_cross_version_persist.py::test_cycle_versions[version_settings2] - chromadb.errors.NotFoundError: Collection [A01] does not exists
Falsifying example: test_cycle_versions(
    version_settings=('1.0.15',
     Settings(environment='', chroma_api_impl='chromadb.api.rust.RustBindingsAPI', chroma_server_nofile=None, chroma_server_thread_pool_size=40, tenant_id='default', topic_namespace='default', chroma_server_host=None, chroma_server_headers=None, chroma_server_http_port=None, chroma_server_ssl_enabled=False, chroma_server_ssl_verify=None, chroma_server_api_default_path=<APIVersion.V2: '/api/v2'>, chroma_server_cors_allow_origins=[], is_persistent=True, persist_directory='/tmp/tmp2_k7jg6k', chroma_memory_limit_bytes=0, chroma_segment_cache_policy=None, allow_reset=True, chroma_auth_token_transport_header=None, chroma_client_auth_provider=None, chroma_client_auth_credentials=None, chroma_server_auth_ignore_paths={'/api/v2': ['GET'], '/api/v2/heartbeat': ['GET'], '/api/v2/version': ['GET'], '/api/v1': ['GET'], '/api/v1/heartbeat': ['GET'], '/api/v1/version': ['GET']}, chroma_overwrite_singleton_tenant_database_access_from_auth=False, chroma_server_authn_provider=None, chroma_server_authn_credentials=None, chroma_server_authn_credentials_file=None, chroma_server_authz_provider=None, chroma_server_authz_config=None, chroma_server_authz_config_file=None, chroma_product_telemetry_impl='chromadb.telemetry.product.posthog.Posthog', chroma_telemetry_impl='chromadb.telemetry.product.posthog.Posthog', anonymized_telemetry=True, chroma_otel_collection_endpoint='', chroma_otel_service_name='chromadb', chroma_otel_collection_headers={}, chroma_otel_granularity=None, migrations='apply', migrations_hash_algorithm='md5', chroma_segment_directory_impl='chromadb.segment.impl.distributed.segment_directory.RendezvousHashSegmentDirectory', chroma_segment_directory_routing_mode=<RoutingMode.ID: 'id'>, chroma_memberlist_provider_impl='chromadb.segment.impl.distributed.segment_directory.CustomResourceMemberlistProvider', worker_memberlist_name='query-service-memberlist', chroma_server_grpc_port=None, chroma_sysdb_impl='chromadb.db.impl.sqlite.SqliteDB', chroma_producer_impl='chromadb.db.impl.sqlite.SqliteDB', chroma_consumer_impl='chromadb.db.impl.sqlite.SqliteDB', chroma_segment_manager_impl='chromadb.segment.impl.manager.local.LocalSegmentManager', chroma_executor_impl='chromadb.execution.executor.local.LocalExecutor', chroma_query_replication_factor=2, chroma_quota_provider_impl=None, chroma_rate_limiting_provider_impl=None, chroma_quota_enforcer_impl='chromadb.quota.simple_quota_enforcer.SimpleQuotaEnforcer', chroma_rate_limit_enforcer_impl='chromadb.rate_limit.simple_rate_limit.SimpleRateLimitEnforcer', chroma_async_rate_limit_enforcer_impl='chromadb.rate_limit.simple_rate_limit.SimpleAsyncRateLimitEnforcer', chroma_logservice_request_timeout_seconds=3, chroma_sysdb_request_timeout_seconds=3, chroma_query_request_timeout_seconds=60, chroma_db_impl=None, chroma_collection_assignment_policy_impl='chromadb.ingest.impl.simple_policy.SimpleAssignmentPolicy', chroma_coordinator_host='localhost', chroma_logservice_host='localhost', chroma_logservice_port=50052)),
    collection_strategy=Collection(name='A01', metadata={'hnsw:construction_ef': 128, 'hnsw:search_ef': 128, 'hnsw:M': 128}, embedding_function=hashing_embedding_function(dim=2, dtype=<class 'numpy.float16'>), id=UUID('2710789e-c49e-4b06-8083-e03cc84f19d4'), dimension=2, dtype=<class 'numpy.float16'>, known_metadata_keys={}, known_document_keywords=[], has_documents=False, has_embeddings=True),
    embeddings_strategy={'ids': ['0'],
     'embeddings': [[0.09765625, 0.430419921875]],
     'metadatas': [None],
     'documents': None},  # or any other generated value

)
You can reproduce this example by temporarily adding @reproduce_failure('6.112.2', b'AXicY2BgZAACMAEGjDAmAABWAAU=') as a decorator on your test case
============== 1 failed, 2 passed, 1 warning in 233.75s (0:03:53) ==============
Error: Process completed with exit code 1.

PR checks / Go tests / cluster-test

Step "Run bin/cluster-test.sh bash -c 'cd go && make test'" from job "Go tests / cluster-test" is failing. The last 20 log lines are:

[...]
testing.tRunner(0xc000dcc000, 0x21dfc50)
	/home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.10.linux-amd64/src/testing/testing.go:1690 +0x227
created by testing.(*T).Run in goroutine 1
	/home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.10.linux-amd64/src/testing/testing.go:1743 +0x826
FAIL	github.com/chroma-core/chroma/go/pkg/sysdb/coordinator	3.440s
	github.com/chroma-core/chroma/go/pkg/sysdb/coordinator/model		coverage: 0.0% of statements
ok  	github.com/chroma-core/chroma/go/pkg/sysdb/grpc	4.465s	coverage: 39.4% of statements
ok  	github.com/chroma-core/chroma/go/pkg/sysdb/metastore/db/dao	5.454s	coverage: 53.3% of statements
	github.com/chroma-core/chroma/go/pkg/sysdb/metastore/db/dao/daotest		coverage: 0.0% of statements
	github.com/chroma-core/chroma/go/pkg/sysdb/metastore/db/dbcore		coverage: 0.0% of statements
	github.com/chroma-core/chroma/go/pkg/sysdb/metastore/db/dbmodel		coverage: 0.0% of statements
	github.com/chroma-core/chroma/go/pkg/sysdb/metastore/db/dbmodel/mocks		coverage: 0.0% of statements
	github.com/chroma-core/chroma/go/pkg/sysdb/metastore/s3		coverage: 0.0% of statements
	github.com/chroma-core/chroma/go/pkg/types		coverage: 0.0% of statements
ok  	github.com/chroma-core/chroma/go/pkg/utils	(cached)	coverage: 31.2% of statements
	github.com/chroma-core/chroma/go/shared/libs		coverage: 0.0% of statements
	github.com/chroma-core/chroma/go/shared/otel		coverage: 0.0% of statements
FAIL
make: *** [Makefile:35: test] Error 1
Error: Process completed with exit code 2.

PR checks / Python tests / test-cluster-rust-frontend (3.9, chromadb/test/property/test_collections.py)

Step "Test" from job "Python tests / test-cluster-rust-frontend (3.9, chromadb/test/property/test_collections.py)" is failing. The last 20 log lines are:

[...]
chromadb/test/property/test_collections.py: 66 warnings
  /home/runner/_work/chroma/chroma/chromadb/api/collection_configuration.py:762: DeprecationWarning: The EmbeddingFunction class does not implement name(). This will be required in a future version.
    embedding_function.name() != "default"

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
============================= slowest 10 durations =============================
115.29s call     chromadb/test/property/test_collections.py::test_collections[basic_http_client]
0.38s setup    chromadb/test/property/test_collections.py::test_collections[basic_http_client]
0.14s setup    chromadb/test/property/test_collections.py::test_previously_failing_two[basic_http_client]
0.13s setup    chromadb/test/property/test_collections.py::test_previously_failing_one[basic_http_client]
0.12s call     chromadb/test/property/test_collections.py::test_previously_failing_one[basic_http_client]
0.03s call     chromadb/test/property/test_collections.py::test_previously_failing_two[basic_http_client]

(3 durations < 0.005s hidden.  Use -vv to show these durations.)
=========================== short test summary info ============================
FAILED chromadb/test/property/test_collections.py::test_collections[basic_http_client] - exceptiongroup.ExceptionGroup: Hypothesis found 2 distinct failures. (2 sub-exceptions)
FAILED chromadb/test/property/test_collections.py::test_previously_failing_one[basic_http_client] - AssertionError
FAILED chromadb/test/property/test_collections.py::test_previously_failing_two[basic_http_client] - AssertionError
================= 3 failed, 1025 warnings in 116.29s (0:01:56) =================
Error: Process completed with exit code 1.

PR checks / Python tests / test-cluster-rust-frontend (3.9, chromadb/test/property/test_collections_with_database_tenant.py)

No logs available for this step.

---

Summary: 1 successful workflow, 1 failed workflow

• ❌ PR checks (46 jobs succeeded, 2 jobs pending, 4 jobs failed)
• ✅ Check PR Title (1 job succeeded)

Last updated: 2025-07-10 20:00:12 UTC

[HammadB]

Before we can implement this in distributed we need to consider if we have the right schema for performance.

[HammadB]

We also need to consider how this would affect future sharding plans, and what indexes we need to support this. Cc @sanketkedia. I think we are jumping to implementation hastily.

[jairad26]

sorry, did not mean to have opened this pr, was just messing around with implementation with claude code. closing

[HammadB]

I mean I think we should do this - just saying we need to consider how to do it