Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
116 commits
Select commit Hold shift + click to select a range
dde898a
starts adding benchmarks
irevoire Mar 22, 2025
ab10e93
fix the benchmarks, the lat/lng required a conversion
irevoire Mar 22, 2025
d767430
let the display crate starts from an already existing database
irevoire Mar 22, 2025
a3abdc2
finish implementing the algorithm, it seems to work?
irevoire Mar 23, 2025
68ea9cc
compute stats about the indexed cells
irevoire Mar 23, 2025
091ded8
display what cells were explored to find the matching points
irevoire Mar 23, 2025
be4b0e1
Move the computations to another threads
irevoire Mar 23, 2025
6202cdf
improve the filtering part to display the steps
irevoire Mar 23, 2025
96ea766
do not display the points that don't fit on screen. It makes the rend…
irevoire Mar 23, 2025
3c43918
add a shape store that doesn't works
irevoire Mar 24, 2025
41c93db
explore the cells from the lower resolution to the higher one
irevoire Mar 24, 2025
90472da
add a filter to ignore some resolutions when analyzing a filter request
irevoire Mar 24, 2025
3e7e533
fix the addition of points
irevoire Mar 25, 2025
55d99af
add a way to filter the DB cells to display
irevoire Apr 5, 2025
9e1464e
add the same slider to the polygon details
irevoire Apr 5, 2025
3a79a2b
get rids of the transmute
irevoire Apr 5, 2025
697e2c8
display the cell associated with where you clicked
irevoire Apr 5, 2025
16010b2
update h3o to the latest version + move some dependencies in workspac…
irevoire Apr 5, 2025
ebcd9b8
use a bounding box instead of a cell when searching
irevoire Apr 8, 2025
dc0ae9b
add an option to display either the h3 cell or the bounding boxes
irevoire Apr 10, 2025
116c4b8
add geojson as a dependency for the future
irevoire Apr 11, 2025
f833ff4
add the bounding box function that was missing for the past 10 commits
irevoire Apr 11, 2025
9ff053d
Add more query benchmarks including real region of france
irevoire Apr 11, 2025
47ba8be
fix warn
irevoire Apr 11, 2025
890ae28
start supporting geojson
irevoire Apr 12, 2025
31a4f5e
fix the display
irevoire Apr 12, 2025
fa95406
never check twice the same cell
irevoire Apr 13, 2025
ede712f
improve the benchmark slightly
irevoire Apr 13, 2025
35a0af7
update the benchmarks with the cadastre
irevoire Apr 13, 2025
976fea0
fix warnings
irevoire Apr 13, 2025
bd7526f
improve search performance
Apr 18, 2025
21174fe
prepared for shape insertion
Apr 18, 2025
3ba2314
fix tests
irevoire Apr 19, 2025
fbb88ec
use geojson::Value directly in visualizer
irevoire Apr 22, 2025
6cfb1c4
add support for the multipoint shape
irevoire Apr 22, 2025
40182dd
prepare the runner to match shape instead of points
irevoire Apr 22, 2025
5530fd8
add support of polygon on the ui
irevoire Apr 22, 2025
5b9f5bc
store shapes in DB
irevoire Apr 22, 2025
3f84190
handle visualization of polygons + polygon filtering
irevoire Apr 22, 2025
90ff740
fix bug when we reach the lowest resolution + use the subshape in rec…
irevoire Apr 22, 2025
5c7a0d4
fmt
irevoire Apr 22, 2025
f8b4f70
get rids of the bounding box options since it’s not used anymore
irevoire Apr 25, 2025
23e876e
provide helpers to draw crosses
irevoire Apr 25, 2025
09e322d
add display db content for multipoints
irevoire Apr 25, 2025
cd4c3a1
stop drawing polygons that don't fit on screen
irevoire Apr 25, 2025
88cdb64
store the names of the documents in the benchmarks
irevoire Apr 27, 2025
ead0aad
add basic support to search in the metadata
irevoire Apr 28, 2025
9d535fa
fmt
irevoire Apr 28, 2025
20e0f0e
add items_inspector
irevoire Apr 28, 2025
03e29f1
typo tolerence
NoodleSamaChan Apr 28, 2025
0519d3e
display selected item on map
NoodleSamaChan Apr 28, 2025
71ad788
add link to geojson website
NoodleSamaChan Apr 28, 2025
ad3c1c0
make the geojson pretty
irevoire Apr 28, 2025
edb0318
stop using a custom syntax hihghlighting thing
irevoire Apr 28, 2025
6114f3b
refactorize slightly
irevoire Apr 28, 2025
c2bc057
display all the cells that contains a shape
irevoire May 1, 2025
57d26e1
make right panel scrollable
irevoire May 1, 2025
310a871
fix duplicate results when inspecting elements
irevoire May 1, 2025
8f8671c
add resolution range filter for inspected items' cells
irevoire May 1, 2025
7c700c7
add copy button in item inspector
irevoire May 1, 2025
177ee18
use the actual ids of the items in the runner instead of relying on t…
irevoire May 2, 2025
889559e
fix id bug when inserting the first new item in an already existing DB
irevoire May 2, 2025
8f3173d
fix the shape support but it's turbo slow
irevoire Jun 1, 2025
050cf2c
add support for multi-polygon
irevoire Jun 8, 2025
89f79b3
add shape benchmark with the french canton
irevoire Jun 8, 2025
197494a
add more benchmarks with parts of fronce
irevoire Jun 8, 2025
d3315e1
add the mix of all the zones
irevoire Jun 8, 2025
e6c9438
Fix the ui to display multi polygon
irevoire Jul 8, 2025
4ed07fc
increase the map size so we can open the cadatre
irevoire Jul 8, 2025
16d533d
compute the query on a cold and hot database
irevoire Jul 8, 2025
26415ca
display the total number of elements
irevoire Jul 8, 2025
7969cc9
rename Writer to Cellulite and add_item to add
irevoire Jul 16, 2025
7a0390d
store the deleted and updated items during a first phase and handle t…
irevoire Jul 16, 2025
a67cae6
big fmt
irevoire Jul 16, 2025
39a0ee9
finish rewriting the whole indexing process
irevoire Jul 16, 2025
ab47fd3
fix the first level 0 iteration before actually indexing
irevoire Jul 16, 2025
5027aaa
fix all the bugs
irevoire Jul 16, 2025
ec5223d
update the benchmarks and visualizer to use the build function
irevoire Jul 16, 2025
6a35796
stop being a coward and use the full french zones :grin:
irevoire Jul 16, 2025
7394715
improve error handling
irevoire Jul 16, 2025
f303e6e
remove commented out code
irevoire Jul 16, 2025
8937896
fix the add and delete methods
irevoire Jul 16, 2025
15b1356
add the progress to cellulite
irevoire Jul 17, 2025
7856a84
add a new cli parameter to the benchmark to limit the total number of…
irevoire Jul 17, 2025
306e002
update steppe again and start adding a huge benchmark, the assets are…
irevoire Jul 18, 2025
f621a75
use another database to hold the updated items
irevoire Jul 18, 2025
bfe16cf
move the errors definition to their own file
irevoire Jul 18, 2025
f9b2189
add two more benchmark file I had forgotten in the previous commits
irevoire Jul 18, 2025
c041ecb
make the test compile again
irevoire Jul 18, 2025
fff966a
update steppe and stop using the default features in the lib
irevoire Jul 18, 2025
c12a904
add the cadastre parcelle dataset in git lfs :tada:
irevoire Jul 18, 2025
944ac1c
fmt
irevoire Jul 18, 2025
f48471b
provide options to skip inserting the documents, commiting or buildin…
irevoire Jul 18, 2025
858379b
update the api of the db handle
irevoire Jul 18, 2025
0fc1e0c
finish moving everything over to git fls maybe
irevoire Jul 18, 2025
8190331
add gitignore
irevoire Jul 18, 2025
67290ea
stop printing the total times twice
irevoire Jul 18, 2025
f865277
improve logging while importing points slightly more
irevoire Jul 18, 2025
cd7e171
handle curvature of earth + display improvements
irevoire Jul 19, 2025
79915c6
integrate zerometry into cellulite, there is a bug left
irevoire Jul 21, 2025
493499a
improve logs and cell display
irevoire Jul 21, 2025
d7c734a
stop writing useless cells
irevoire Jul 21, 2025
9cd8d6b
fix the integration with zerometry
irevoire Jul 22, 2025
978c99c
prepare for integration within meilisearch
irevoire Jul 23, 2025
a798373
ignore the assets directory when importing the lib
irevoire Jul 23, 2025
53ee3f8
prepare for publish
irevoire Jul 23, 2025
1edf3c6
Provide a function to add raw zerometry without going from the geojson
irevoire Jul 24, 2025
127f7c9
prepare for release
irevoire Jul 24, 2025
cbc76bf
stop blocking everything when the fst is stuck
irevoire Jul 25, 2025
cd7166e
make clippy happy
irevoire Jul 27, 2025
d5f7605
use different DB to store items and cells
irevoire Jul 27, 2025
ae13661
divide indexing time by two by keeping a map of the items instead of …
irevoire Jul 27, 2025
fe0262e
use an intmap instead of an hashmap for the frozen reader
irevoire Jul 27, 2025
3bcece1
parallelize the item insertion to the level-0
irevoire Jul 27, 2025
2490796
make the retrieval of frozen items faster by specifying the capacity …
irevoire Jul 28, 2025
28b860b
WIP
irevoire Aug 14, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .gitattributes
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
*.geojson.gz filter=lfs diff=lfs merge=lfs -text
*.json.gz filter=lfs diff=lfs merge=lfs -text
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
*.mdb
*.target
3 changes: 3 additions & 0 deletions .gitmodules
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
[submodule "assets/france-geojson"]
path = assets/france-geojson
url = https://github.com/gregoiredavid/france-geojson
41 changes: 34 additions & 7 deletions Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,19 +1,46 @@
[package]
name = "cellulite"
version = "0.1.0"
edition = "2021"
version = "0.1.1"
edition = "2024"
license-file = "LICENSE"
description = "Store and retrieve geojson in a memory mapped database"
documentation = "https://docs.rs/cellulite"
repository = "https://github.com/irevoire/cellulite"
authors = ["Tamo <tamo@meilisearch.com>"]
keywords = ["geo", "geography", "database"]

exclude = ["assets"]

[workspace]
members = ["examples/*"]
members = [ "benchmarks","examples/*"]

[dependencies]
geo = "0.29.3"
[workspace.dependencies]
geo = "0.30.0"
geojson = "0.24.2"
geo-types = "0.7.15"
h3o = { version = "0.7.1", features = ["geo"] }
h3o = { version = "0.8.0", features = ["geo"] }
heed = "0.22.0"
ordered-float = "5.0.0"
roaring = "0.10.10"
roaring = "0.11.1"
thiserror = "2.0.12"
steppe = { version = "0.4.0", default-features = false }
zerometry = { version = "0.1.0" }

[dependencies]
geo = { workspace = true }
geo-types = { workspace = true }
geojson = { workspace = true }
h3o = { workspace = true }
heed = { workspace = true }
ordered-float = { workspace = true }
roaring = { workspace = true }
thiserror = { workspace = true }
steppe = { workspace = true }
zerometry = { workspace = true }
intmap = "3.1.2"
rayon = "1.10.0"
crossbeam = "0.8.4"
thread_local = "1.1.9"

[dev-dependencies]
insta = "1.42.2"
Expand Down
13 changes: 13 additions & 0 deletions LICENSE
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE
Version 2, December 2004

Copyright (C) 2004 Sam Hocevar <sam@hocevar.net>

Everyone is permitted to copy and distribute verbatim or modified
copies of this license document, and changing it is allowed as long
as the name is changed.

DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE
TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION

0. You just DO WHAT THE FUCK YOU WANT TO.
3 changes: 3 additions & 0 deletions assets/cadastre/adresses-cadastre-01.geojson.gz
Git LFS file not shown
3 changes: 3 additions & 0 deletions assets/cadastre/adresses-cadastre-02.geojson.gz
Git LFS file not shown
3 changes: 3 additions & 0 deletions assets/cadastre/adresses-cadastre-03.geojson.gz
Git LFS file not shown
3 changes: 3 additions & 0 deletions assets/cadastre/adresses-cadastre-04.geojson.gz
Git LFS file not shown
3 changes: 3 additions & 0 deletions assets/cadastre/adresses-cadastre-05.geojson.gz
Git LFS file not shown
3 changes: 3 additions & 0 deletions assets/cadastre/adresses-cadastre-06.geojson.gz
Git LFS file not shown
3 changes: 3 additions & 0 deletions assets/cadastre/adresses-cadastre-07.geojson.gz
Git LFS file not shown
3 changes: 3 additions & 0 deletions assets/cadastre/adresses-cadastre-08.geojson.gz
Git LFS file not shown
3 changes: 3 additions & 0 deletions assets/cadastre/adresses-cadastre-09.geojson.gz
Git LFS file not shown
3 changes: 3 additions & 0 deletions assets/cadastre/adresses-cadastre-10.geojson.gz
Git LFS file not shown
3 changes: 3 additions & 0 deletions assets/cadastre/adresses-cadastre-11.geojson.gz
Git LFS file not shown
3 changes: 3 additions & 0 deletions assets/cadastre/adresses-cadastre-12.geojson.gz
Git LFS file not shown
3 changes: 3 additions & 0 deletions assets/cadastre/adresses-cadastre-13.geojson.gz
Git LFS file not shown
3 changes: 3 additions & 0 deletions assets/cadastre/adresses-cadastre-14.geojson.gz
Git LFS file not shown
3 changes: 3 additions & 0 deletions assets/cadastre/adresses-cadastre-15.geojson.gz
Git LFS file not shown
3 changes: 3 additions & 0 deletions assets/cadastre/adresses-cadastre-16.geojson.gz
Git LFS file not shown
3 changes: 3 additions & 0 deletions assets/cadastre/adresses-cadastre-17.geojson.gz
Git LFS file not shown
3 changes: 3 additions & 0 deletions assets/cadastre/adresses-cadastre-18.geojson.gz
Git LFS file not shown
3 changes: 3 additions & 0 deletions assets/cadastre/adresses-cadastre-19.geojson.gz
Git LFS file not shown
3 changes: 3 additions & 0 deletions assets/cadastre/adresses-cadastre-21.geojson.gz
Git LFS file not shown
3 changes: 3 additions & 0 deletions assets/cadastre/adresses-cadastre-22.geojson.gz
Git LFS file not shown
3 changes: 3 additions & 0 deletions assets/cadastre/adresses-cadastre-23.geojson.gz
Git LFS file not shown
3 changes: 3 additions & 0 deletions assets/cadastre/adresses-cadastre-24.geojson.gz
Git LFS file not shown
3 changes: 3 additions & 0 deletions assets/cadastre/adresses-cadastre-25.geojson.gz
Git LFS file not shown
3 changes: 3 additions & 0 deletions assets/cadastre/adresses-cadastre-26.geojson.gz
Git LFS file not shown
3 changes: 3 additions & 0 deletions assets/cadastre/adresses-cadastre-27.geojson.gz
Git LFS file not shown
3 changes: 3 additions & 0 deletions assets/cadastre/adresses-cadastre-28.geojson.gz
Git LFS file not shown
3 changes: 3 additions & 0 deletions assets/cadastre/adresses-cadastre-29.geojson.gz
Git LFS file not shown
3 changes: 3 additions & 0 deletions assets/cadastre/adresses-cadastre-30.geojson.gz
Git LFS file not shown
3 changes: 3 additions & 0 deletions assets/cadastre/adresses-cadastre-31.geojson.gz
Git LFS file not shown
3 changes: 3 additions & 0 deletions assets/cadastre/adresses-cadastre-32.geojson.gz
Git LFS file not shown
3 changes: 3 additions & 0 deletions assets/cadastre/adresses-cadastre-33.geojson.gz
Git LFS file not shown
3 changes: 3 additions & 0 deletions assets/cadastre/adresses-cadastre-34.geojson.gz
Git LFS file not shown
3 changes: 3 additions & 0 deletions assets/cadastre/adresses-cadastre-35.geojson.gz
Git LFS file not shown
3 changes: 3 additions & 0 deletions assets/cadastre/adresses-cadastre-36.geojson.gz
Git LFS file not shown
3 changes: 3 additions & 0 deletions assets/cadastre/adresses-cadastre-37.geojson.gz
Git LFS file not shown
3 changes: 3 additions & 0 deletions assets/cadastre/adresses-cadastre-38.geojson.gz
Git LFS file not shown
3 changes: 3 additions & 0 deletions assets/cadastre/adresses-cadastre-39.geojson.gz
Git LFS file not shown
3 changes: 3 additions & 0 deletions assets/cadastre/adresses-cadastre-40.geojson.gz
Git LFS file not shown
3 changes: 3 additions & 0 deletions assets/cadastre/adresses-cadastre-41.geojson.gz
Git LFS file not shown
3 changes: 3 additions & 0 deletions assets/cadastre/adresses-cadastre-42.geojson.gz
Git LFS file not shown
3 changes: 3 additions & 0 deletions assets/cadastre/adresses-cadastre-43.geojson.gz
Git LFS file not shown
3 changes: 3 additions & 0 deletions assets/cadastre/adresses-cadastre-44.geojson.gz
Git LFS file not shown
3 changes: 3 additions & 0 deletions assets/cadastre/adresses-cadastre-45.geojson.gz
Git LFS file not shown
3 changes: 3 additions & 0 deletions assets/cadastre/adresses-cadastre-46.geojson.gz
Git LFS file not shown
3 changes: 3 additions & 0 deletions assets/cadastre/adresses-cadastre-47.geojson.gz
Git LFS file not shown
3 changes: 3 additions & 0 deletions assets/cadastre/adresses-cadastre-48.geojson.gz
Git LFS file not shown
3 changes: 3 additions & 0 deletions assets/cadastre/adresses-cadastre-49.geojson.gz
Git LFS file not shown
3 changes: 3 additions & 0 deletions assets/cadastre/adresses-cadastre-50.geojson.gz
Git LFS file not shown
3 changes: 3 additions & 0 deletions assets/cadastre/adresses-cadastre-51.geojson.gz
Git LFS file not shown
3 changes: 3 additions & 0 deletions assets/cadastre/adresses-cadastre-52.geojson.gz
Git LFS file not shown
3 changes: 3 additions & 0 deletions assets/cadastre/adresses-cadastre-53.geojson.gz
Git LFS file not shown
3 changes: 3 additions & 0 deletions assets/cadastre/adresses-cadastre-54.geojson.gz
Git LFS file not shown
3 changes: 3 additions & 0 deletions assets/cadastre/adresses-cadastre-55.geojson.gz
Git LFS file not shown
3 changes: 3 additions & 0 deletions assets/cadastre/adresses-cadastre-56.geojson.gz
Git LFS file not shown
3 changes: 3 additions & 0 deletions assets/cadastre/adresses-cadastre-57.geojson.gz
Git LFS file not shown
3 changes: 3 additions & 0 deletions assets/cadastre/adresses-cadastre-58.geojson.gz
Git LFS file not shown
3 changes: 3 additions & 0 deletions assets/cadastre/adresses-cadastre-59.geojson.gz
Git LFS file not shown
3 changes: 3 additions & 0 deletions assets/cadastre/adresses-cadastre-60.geojson.gz
Git LFS file not shown
3 changes: 3 additions & 0 deletions assets/cadastre/adresses-cadastre-61.geojson.gz
Git LFS file not shown
3 changes: 3 additions & 0 deletions assets/cadastre/adresses-cadastre-62.geojson.gz
Git LFS file not shown
3 changes: 3 additions & 0 deletions assets/cadastre/adresses-cadastre-63.geojson.gz
Git LFS file not shown
3 changes: 3 additions & 0 deletions assets/cadastre/adresses-cadastre-64.geojson.gz
Git LFS file not shown
3 changes: 3 additions & 0 deletions assets/cadastre/adresses-cadastre-65.geojson.gz
Git LFS file not shown
3 changes: 3 additions & 0 deletions assets/cadastre/adresses-cadastre-66.geojson.gz
Git LFS file not shown
3 changes: 3 additions & 0 deletions assets/cadastre/adresses-cadastre-67.geojson.gz
Git LFS file not shown
3 changes: 3 additions & 0 deletions assets/cadastre/adresses-cadastre-68.geojson.gz
Git LFS file not shown
3 changes: 3 additions & 0 deletions assets/cadastre/adresses-cadastre-69.geojson.gz
Git LFS file not shown
3 changes: 3 additions & 0 deletions assets/cadastre/adresses-cadastre-70.geojson.gz
Git LFS file not shown
3 changes: 3 additions & 0 deletions assets/cadastre/adresses-cadastre-71.geojson.gz
Git LFS file not shown
3 changes: 3 additions & 0 deletions assets/cadastre/adresses-cadastre-72.geojson.gz
Git LFS file not shown
3 changes: 3 additions & 0 deletions assets/cadastre/adresses-cadastre-73.geojson.gz
Git LFS file not shown
3 changes: 3 additions & 0 deletions assets/cadastre/adresses-cadastre-74.geojson.gz
Git LFS file not shown
3 changes: 3 additions & 0 deletions assets/cadastre/adresses-cadastre-75.geojson.gz
Git LFS file not shown
3 changes: 3 additions & 0 deletions assets/cadastre/adresses-cadastre-76.geojson.gz
Git LFS file not shown
3 changes: 3 additions & 0 deletions assets/cadastre/adresses-cadastre-77.geojson.gz
Git LFS file not shown
3 changes: 3 additions & 0 deletions assets/cadastre/adresses-cadastre-78.geojson.gz
Git LFS file not shown
3 changes: 3 additions & 0 deletions assets/cadastre/adresses-cadastre-79.geojson.gz
Git LFS file not shown
3 changes: 3 additions & 0 deletions assets/cadastre/adresses-cadastre-80.geojson.gz
Git LFS file not shown
3 changes: 3 additions & 0 deletions assets/cadastre/adresses-cadastre-81.geojson.gz
Git LFS file not shown
Loading