Skip to content

Add an example of embedding indexes inside a parquet file #16395

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 28 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 24 commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
793abb9
Add an example of embedding indexes inside a parquet file
zhuqi-lucas Jun 13, 2025
1f480ee
Add page image
zhuqi-lucas Jun 13, 2025
2a0ecac
Add prune file example
zhuqi-lucas Jun 13, 2025
4e61d0e
Fix clippy
zhuqi-lucas Jun 13, 2025
a8da658
polish code
zhuqi-lucas Jun 13, 2025
c1ab4b9
Fmt
zhuqi-lucas Jun 13, 2025
baf0311
address comments
zhuqi-lucas Jun 13, 2025
18e7028
Add debug
zhuqi-lucas Jun 13, 2025
66dc5e4
Add new example, but it will fail with page index
zhuqi-lucas Jun 14, 2025
eb9b62e
add debug
zhuqi-lucas Jun 14, 2025
310576e
add debug
zhuqi-lucas Jun 14, 2025
fbeecfe
polish
zhuqi-lucas Jun 15, 2025
88fc6a6
debug
zhuqi-lucas Jun 15, 2025
32abcb9
Using low level API to support
zhuqi-lucas Jun 18, 2025
284510c
polish
zhuqi-lucas Jun 18, 2025
0410bd8
fix
zhuqi-lucas Jun 18, 2025
56ad7f6
Merge remote-tracking branch 'upstream/main' into embedding_indexes
zhuqi-lucas Jun 18, 2025
12ce9c2
merge
zhuqi-lucas Jun 18, 2025
0c093ac
fix
zhuqi-lucas Jun 18, 2025
a789084
Merge remote-tracking branch 'upstream/main' into embedding_indexes
zhuqi-lucas Jun 18, 2025
9c75814
complte solution
zhuqi-lucas Jun 19, 2025
13c1706
Merge remote-tracking branch 'upstream/main' into embedding_indexes
zhuqi-lucas Jun 19, 2025
06d6f08
polish comments
zhuqi-lucas Jun 19, 2025
6bd7d3e
adjust image
zhuqi-lucas Jun 19, 2025
23d7125
add comments part 1
zhuqi-lucas Jun 20, 2025
13b74ac
pin to new arrow-rs
zhuqi-lucas Jun 21, 2025
1b0501c
pin to new arrow-rs
zhuqi-lucas Jun 21, 2025
c344843
add comments part 2
zhuqi-lucas Jun 21, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 16 additions & 33 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

14 changes: 7 additions & 7 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -88,19 +88,19 @@ ahash = { version = "0.8", default-features = false, features = [
"runtime-rng",
] }
apache-avro = { version = "0.17", default-features = false }
arrow = { version = "55.1.0", features = [
arrow = { git = "https://github.com/zhuqi-lucas/arrow-rs.git", branch = "add_pub_write_all_api", features = [
"prettyprint",
"chrono-tz",
] }
arrow-buffer = { version = "55.0.0", default-features = false }
arrow-flight = { version = "55.1.0", features = [
arrow-buffer = { git = "https://github.com/zhuqi-lucas/arrow-rs.git", branch = "add_pub_write_all_api", default-features = false }
arrow-flight = { git = "https://github.com/zhuqi-lucas/arrow-rs.git", branch = "add_pub_write_all_api", features = [
"flight-sql-experimental",
] }
arrow-ipc = { version = "55.0.0", default-features = false, features = [
arrow-ipc = { git = "https://github.com/zhuqi-lucas/arrow-rs.git", branch = "add_pub_write_all_api", default-features = false, features = [
"lz4",
] }
arrow-ord = { version = "55.0.0", default-features = false }
arrow-schema = { version = "55.0.0", default-features = false }
arrow-ord = { git = "https://github.com/zhuqi-lucas/arrow-rs.git", branch = "add_pub_write_all_api", default-features = false }
arrow-schema = { git = "https://github.com/zhuqi-lucas/arrow-rs.git", branch = "add_pub_write_all_api", default-features = false }
async-trait = "0.1.88"
bigdecimal = "0.4.8"
bytes = "1.10"
Expand Down Expand Up @@ -152,7 +152,7 @@ itertools = "0.14"
log = "^0.4"
object_store = { version = "0.12.0", default-features = false }
parking_lot = "0.12"
parquet = { version = "55.1.0", default-features = false, features = [
parquet = { git = "https://github.com/zhuqi-lucas/arrow-rs.git", branch = "add_pub_write_all_api", default-features = false, features = [
"arrow",
"async",
"object_store",
Expand Down
Loading
Loading