Skip to content

Document Table Constraint Enforcement Behavior in Custom Table Providers Guide #16340

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
Jun 12, 2025
Merged

Document Table Constraint Enforcement Behavior in Custom Table Providers Guide #16340

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
eb9fff5
Add documentation for table constraint enforcement in DataFusion
kosiew Jun 7, 2025
6d3b5c4
Merge branch 'main' into doc-16309
kosiew Jun 9, 2025
6fd4527
Merge branch 'main' into doc-16309
kosiew Jun 9, 2025
d6d51db
Add link to table constraints documentation in index.rst
kosiew Jun 9, 2025
3a9ee8d
fix: correct markdown link references for table constraints documenta…
kosiew Jun 9, 2025
46859f2
Update docs/source/library-user-guide/table-constraints.md
kosiew Jun 11, 2025
27e396c
fix: remove incorrect information about optimizer constraints in tabl…
kosiew Jun 11, 2025
8da0cbe
prettier fix
kosiew Jun 11, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/source/index.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -135,6 +135,7 @@ To get started, see
library-user-guide/catalogs
library-user-guide/adding-udfs
library-user-guide/custom-table-providers
library-user-guide/table-constraints
library-user-guide/extending-operators
library-user-guide/profiling
library-user-guide/query-optimizer
Expand Down
3 changes: 3 additions & 0 deletions docs/source/library-user-guide/custom-table-providers.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,9 @@ Like other areas of DataFusion, you extend DataFusion's functionality by impleme

This section describes how to create a [`TableProvider`] and how to configure DataFusion to use it for reading.

For details on how table constraints such as primary keys or unique
constraints are handled, see [Table Constraint Enforcement](table-constraints.md).

## Table Provider and Scan

The [`TableProvider::scan`] method reads data from the table and is likely the most important. It returns an [`ExecutionPlan`] that DataFusion will use to read the actual data during execution of the query. The [`TableProvider::insert_into`] method is used to `INSERT` data into the table.
Expand Down
46 changes: 46 additions & 0 deletions docs/source/library-user-guide/table-constraints.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
<!---
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->

# Table Constraint Enforcement

Table providers can describe table constraints using the
[`TableConstraint`] and [`Constraints`] APIs. These constraints include
primary keys, unique keys, foreign keys and check constraints.

DataFusion does **not** currently enforce these constraints at runtime.
They are provided for informational purposes and can be used by custom
`TableProvider` implementations or other parts of the system.

- **Nullability**: The only property enforced by DataFusion is the
nullability of each [`Field`] in a schema. Columns marked as not
nullable should not produce null values during execution. DataFusion
does not check this when data is ingested.
- **Primary and unique keys**: DataFusion does not verify that the data
satisfies primary or unique key constraints. Table providers that
require this behaviour must implement their own checks.
- **Foreign keys and check constraints**: These constraints are parsed
but are not validated or used during query planning.

The optimizer also does not assume that these constraints hold when
rewriting queries. For example, declaring a column as a primary key will
not allow the optimizer to skip a `DISTINCT` aggregation.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't think this was true -- I was pretty sure there are some ordering / functional dependency check that relies on declared constraints, but I couldn't find it quickly when searching

Maybe @mustafasrepo remembers 🤔

Copy link
Contributor Author

@kosiew kosiew Jun 11, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hi @alamb,

You're right.

I tested this in datafusion-cli

-- Test 1: Create table with more data to see if DISTINCT appears
CREATE TABLE test_pk_large (
    id INTEGER PRIMARY KEY,
    name VARCHAR(50)
);

-- Insert duplicate names but unique IDs
INSERT INTO test_pk_large VALUES 
    (1, 'Alice'),
    (2, 'Alice'),
    (3, 'Bob'),
    (4, 'Bob'),
    (5, 'Charlie');

-- Test DISTINCT on primary key column
EXPLAIN SELECT DISTINCT id FROM test_pk_large;

+---------------+-------------------------------+
| plan_type     | plan                          |
+---------------+-------------------------------+
| physical_plan | ┌───────────────────────────┐ |
|               | │       DataSourceExec      │ |
|               | │    --------------------   │ |
|               | │         bytes: 376        │ |
|               | │       format: memory      │ |
|               | │          rows: 1          │ |
|               | └───────────────────────────┘ |
|               |                               |
+---------------+-------------------------------+

-- Test 2
CREATE TABLE test_no_pk (
    id INTEGER,
    name VARCHAR(50)
);

-- Insert unique IDs (same as before)
INSERT INTO test_no_pk VALUES 
    (1, 'Alice'),
    (2, 'Alice'),
    (3, 'Bob'),
    (4, 'Bob'),
    (5, 'Charlie');

EXPLAIN SELECT DISTINCT id FROM test_no_pk;

+---------------+-------------------------------+
| plan_type     | plan                          |
+---------------+-------------------------------+
| physical_plan | ┌───────────────────────────┐ |
|               | │       AggregateExec       │ |
|               | │    --------------------   │ |
|               | │        group_by: id       │ |
|               | │                           │ |
|               | │           mode:           │ |
|               | │      FinalPartitioned     │ |
|               | └─────────────┬─────────────┘ |
|               | ┌─────────────┴─────────────┐ |
|               | │    CoalesceBatchesExec    │ |
|               | │    --------------------   │ |
|               | │     target_batch_size:    │ |
|               | │            8192           │ |
|               | └─────────────┬─────────────┘ |
|               | ┌─────────────┴─────────────┐ |
|               | │      RepartitionExec      │ |
|               | │    --------------------   │ |
|               | │ partition_count(in->out): │ |
|               | │          10 -> 10         │ |
|               | │                           │ |
|               | │    partitioning_scheme:   │ |
|               | │      Hash([id@0], 10)     │ |
|               | └─────────────┬─────────────┘ |
|               | ┌─────────────┴─────────────┐ |
|               | │      RepartitionExec      │ |
|               | │    --------------------   │ |
|               | │ partition_count(in->out): │ |
|               | │          1 -> 10          │ |
|               | │                           │ |
|               | │    partitioning_scheme:   │ |
|               | │    RoundRobinBatch(10)    │ |
|               | └─────────────┬─────────────┘ |
|               | ┌─────────────┴─────────────┐ |
|               | │       AggregateExec       │ |
|               | │    --------------------   │ |
|               | │        group_by: id       │ |
|               | │       mode: Partial       │ |
|               | └─────────────┬─────────────┘ |
|               | ┌─────────────┴─────────────┐ |
|               | │       DataSourceExec      │ |
|               | │    --------------------   │ |
|               | │         bytes: 376        │ |
|               | │       format: memory      │ |
|               | │          rows: 1          │ |
|               | └───────────────────────────┘ |
|               |                               |
+---------------+-------------------------------+

In other words, the declared constraints does affect the optimizer.
I'll remove this paragraph.


[`tableconstraint`]: https://docs.rs/datafusion/latest/datafusion/sql/planner/enum.TableConstraint.html
[`constraints`]: https://docs.rs/datafusion/latest/datafusion/common/functional_dependencies/struct.Constraints.html
[`field`]: https://docs.rs/arrow/latest/arrow/datatype/struct.Field.html