Skip to content

Implemented Durable Graph Database Providers ( golem:graph WIT interfaces) #40

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 87 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 35 commits
Commits
Show all changes
87 commits
Select commit Hold shift + click to select a range
2e3aaaa
implemented graph databases
harshtech123 Jun 17, 2025
a7dc6f7
tests updated
harshtech123 Jun 22, 2025
2cc1f9f
Merge branch 'golemcloud:main' into main
harshtech123 Jun 22, 2025
917205f
bindings
harshtech123 Jun 22, 2025
e614a41
initialize wasm test component for graph
harshtech123 Jun 22, 2025
6186a77
updating wasm tests
harshtech123 Jun 22, 2025
b9a2575
test update
harshtech123 Jun 22, 2025
3fe7fc1
golem integeration testing completed
harshtech123 Jun 24, 2025
ad7c993
clippy fix
harshtech123 Jun 24, 2025
2cfe4ad
conflic resolve
harshtech123 Jun 25, 2025
be616c1
resolve the conflicts
harshtech123 Jun 25, 2025
94dd093
Merge branch 'golemcloud:main' into main
harshtech123 Jun 25, 2025
cf41f9f
refactored according to new project structure
harshtech123 Jun 25, 2025
ded9b07
Merge branch 'golemcloud:main' into main
harshtech123 Jun 25, 2025
3407f6c
durability improvements including feature flags
harshtech123 Jun 25, 2025
029c3ea
comment updated
harshtech123 Jun 25, 2025
a8dd167
clippy and fmt
harshtech123 Jun 25, 2025
41a008c
upgraded error handling
harshtech123 Jun 30, 2025
5b9d585
Merge branch 'golemcloud:main' into main
harshtech123 Jul 18, 2025
8944622
Merge branch 'golemcloud:main' into main
harshtech123 Jul 18, 2025
c31645c
logging
harshtech123 Jul 19, 2025
347289b
logging
harshtech123 Jul 20, 2025
8de5b2a
fmt + clippy
harshtech123 Jul 20, 2025
3dced22
conflict resolve
harshtech123 Jul 20, 2025
6706104
Merge branch 'main' into main
harshtech123 Jul 20, 2025
8d50675
fix build + fmt
harshtech123 Jul 20, 2025
01c7656
makefile
harshtech123 Jul 20, 2025
e543383
fixed
harshtech123 Jul 20, 2025
f41ec4a
Merge branch 'main' into main
harshtech123 Jul 23, 2025
7800c05
logging arangodb
harshtech123 Jul 23, 2025
2ddeaa6
janusgraph logging
harshtech123 Jul 23, 2025
48d1781
neo4j
harshtech123 Jul 23, 2025
24dda4e
conflict resolve
harshtech123 Jul 23, 2025
322f070
Merge branch 'main' into main
harshtech123 Jul 23, 2025
bff4055
cargo.lock
harshtech123 Jul 23, 2025
7efb45a
cargo.toml cleanup
harshtech123 Jul 24, 2025
9e5a350
using log !
harshtech123 Jul 24, 2025
359ff76
fmt
harshtech123 Jul 24, 2025
aac6260
arangodb_error
harshtech123 Jul 24, 2025
f5cef2c
removed unused helpers
harshtech123 Jul 24, 2025
dc9688d
error.rs updated
harshtech123 Jul 24, 2025
41d5fa9
neo4j specific error
harshtech123 Jul 24, 2025
7a3f9f2
commenting //
harshtech123 Jul 24, 2025
cb16a51
upgraded provider specific error status
harshtech123 Jul 24, 2025
96f6794
fmt
harshtech123 Jul 24, 2025
60efa3a
Merge branch 'golemcloud:main' into main
harshtech123 Jul 24, 2025
1e4fa4c
refactor new test structure
harshtech123 Jul 24, 2025
c00d941
golem.yaml and cargo.toml updated
harshtech123 Jul 24, 2025
e85f7d9
makefile update
harshtech123 Jul 24, 2025
3abc877
refactored: cargo.lock
harshtech123 Jul 24, 2025
bcd6a4e
test/graph-lib.rs
harshtech123 Jul 25, 2025
c2d46a5
lib.rs
harshtech123 Jul 25, 2025
3fd51bc
new durability : logging state
harshtech123 Jul 25, 2025
f469965
fmt
harshtech123 Jul 25, 2025
01b2910
wrapper removed !
harshtech123 Jul 25, 2025
08d4ce8
removed unused commented code
harshtech123 Jul 25, 2025
b2d212a
CI for graph
harshtech123 Jul 25, 2025
442c14d
fmt
harshtech123 Jul 25, 2025
a0a3a5e
janusgraph docker
harshtech123 Jul 25, 2025
c9dae41
arangodb database
harshtech123 Jul 25, 2025
e4d0139
arangodb docker
harshtech123 Jul 25, 2025
03791fe
using wit-bindgen 0.36
harshtech123 Jul 25, 2025
f3988a2
arangodb database fix
harshtech123 Jul 25, 2025
58d6148
arangodb collections
harshtech123 Jul 25, 2025
f5a9454
fixed lib.rs
harshtech123 Jul 25, 2025
0c89320
conflict cargo.lock
harshtech123 Jul 25, 2025
9fc8a43
Merge branch 'main' into main
harshtech123 Jul 25, 2025
d17001e
conflict fix
harshtech123 Jul 25, 2025
635f3cb
chrono / removed iso
harshtech123 Jul 25, 2025
fed757d
using with_config_key !
harshtech123 Jul 25, 2025
9cf63a0
fmt
harshtech123 Jul 25, 2025
221cf0d
wit get-schema-manager
harshtech123 Jul 25, 2025
eca13df
lib.rs update for get-schema-manager
harshtech123 Jul 25, 2025
aa18807
removed string based error handling !
harshtech123 Jul 25, 2025
c8c5fe9
some comment and error handling update
harshtech123 Jul 25, 2025
f5389f6
result pushing !
harshtech123 Jul 25, 2025
ee6f878
removed specific handling for janusgraph
harshtech123 Jul 25, 2025
69dc107
fixes for janusgraph
harshtech123 Jul 26, 2025
dd8c634
fmt
harshtech123 Jul 26, 2025
b5f8e93
fmt clippy
harshtech123 Jul 26, 2025
e46c109
lib.rs updated no specific provider handling in this file
harshtech123 Jul 26, 2025
15323cc
cleanup for lib.rs
harshtech123 Jul 26, 2025
911bcbb
error handling !
harshtech123 Jul 26, 2025
cac1802
clippy
harshtech123 Jul 26, 2025
1974b7c
fmt
harshtech123 Jul 26, 2025
85c8eb7
Merge branch 'main' into main
harshtech123 Jul 28, 2025
68482db
refresh
harshtech123 Jul 28, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
98 changes: 98 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

7 changes: 6 additions & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,11 @@ members = [
"search/algolia",
"search/meilisearch",
"search/opensearch",
"search/typesense"
"search/typesense",
"graph/graph",
"graph/arangodb",
"graph/janusgraph",
"graph/neo4j"
]

[profile.release]
Expand All @@ -30,6 +34,7 @@ opt-level = 's'
[workspace.dependencies]
golem-llm = { path = "llm/llm", version = "0.0.0", default-features = false }
golem-search = { path = "search/search", version = "0.0.0", default-features = false}
golem-graph = { path = "graph/graph" , version = "0.0.0" ,default-features =false}
golem-rust = "1.6.0"
log = "0.4.27"
reqwest = { git = "https://github.com/golemcloud/reqwest", branch = "update-may-2025", features = [
Expand Down
27 changes: 8 additions & 19 deletions Makefile.toml
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ args = ["test"]
script_runner = "@duckscript"
script = '''

domains = array llm websearch search
domains = array llm websearch search graph

# if there is no domain passed run for every domain
if is_empty ${1}
Expand All @@ -30,7 +30,7 @@ end
script_runner = "@duckscript"
script = '''

domains = array llm websearch search
domains = array llm websearch search graph

# if there is no domain passed run for every domain
if is_empty ${1}
Expand All @@ -46,9 +46,7 @@ end
script_runner = "@duckscript"
script = '''
#!/bin/bash

domains = array llm websearch search

domains = array llm websearch search graph

# if there is no domain passed run for every domain
if is_empty ${1}
Expand All @@ -64,9 +62,7 @@ end
script_runner = "@duckscript"
script = '''
#!/bin/bash

domains = array llm websearch search

domains = array llm websearch search graph

# if there is no domain passed run for every domain
if is_empty ${1}
Expand All @@ -81,9 +77,7 @@ end
[tasks.wit]
script_runner = "@duckscript"
script = '''

domains = array llm websearch search

domains = array llm websearch search graph

# if there is no domain passed run for every domain
if is_empty ${1}
Expand All @@ -99,9 +93,7 @@ end
description = "Builds all test components with golem-cli"
script_runner = "@duckscript"
script = '''

domains = array llm websearch search

domains = array llm websearch search graph

# if there is no domain passed run for every domain
if is_empty ${1}
Expand Down Expand Up @@ -155,8 +147,7 @@ environment = get_env "ENVIRONMENT"
is_dev = eq ${environment} "dev"


targets = array llm_openai llm_anthropic llm_grok llm_openrouter llm_ollama web_search_brave web_search_google web_search_serper web_search_tavily search_algolia search_elasticsearch search_meilisearch search_opensearch search_typesense

targets = array llm_openai llm_anthropic llm_grok llm_openrouter llm_ollama web_search_brave web_search_google web_search_serper web_search_tavily search_algolia search_elasticsearch search_meilisearch search_opensearch search_typesense graph_arangodb graph_janusgraph graph_neo4j

for target in ${targets}
if ${is_portable}
Expand Down Expand Up @@ -185,9 +176,7 @@ environment = get_env "ENVIRONMENT"
is_dev = eq ${environment} "dev"



targets = array llm_openai llm_anthropic llm_grok llm_openrouter llm_ollama websearch_brave websearch_google websearch_serper websearch_tavily search_algolia search_elasticsearch search_meilisearch search_opensearch search_typesense

targets = array llm_openai llm_anthropic llm_grok llm_openrouter llm_ollama websearch_brave websearch_google websearch_serper websearch_tavily search_algolia search_elasticsearch search_meilisearch search_opensearch search_typesense graph_arangodb graph_janusgraph graph_neo4j

for target in ${targets}
if ${is_portable}
Expand Down
154 changes: 154 additions & 0 deletions graph/Makefile.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,154 @@
[config]
default_to_workspace = false
skip_core_tasks = true

[tasks.build]
run_task = { name = [
"build-arangodb",
"build-janusgraph",
"build-neo4j",
] }

[tasks.build-portable]
run_task = { name = [
"build-arangodb-portable",
"build-janusgraph-portable",
"build-neo4j-portable",
] }

[tasks.release-build]
run_task = { name = [
"release-build-arangodb",
"release-build-janusgraph",
"release-build-neo4j",
] }

[tasks.release-build-portable]
run_task = { name = [
"release-build-arangodb-portable",
"release-build-janusgraph-portable",
"release-build-neo4j-portable",
] }

[tasks.build-arangodb]
install_crate = { crate_name = "cargo-component", version = "0.20.0" }
command = "cargo-component"
args = ["build", "-p", "golem-graph-arangodb"]

[tasks.build-arangodb-portable]
install_crate = { crate_name = "cargo-component", version = "0.20.0" }
command = "cargo-component"
args = ["build", "-p", "golem-graph-arangodb", "--no-default-features"]


[tasks.build-janusgraph]
install_crate = { crate_name = "cargo-component", version = "0.20.0" }
command = "cargo-component"
args = ["build", "-p", "golem-graph-janusgraph"]

[tasks.build-janusgraph-portable]
install_crate = { crate_name = "cargo-component", version = "0.20.0" }
command = "cargo-component"
args = ["build", "-p", "golem-graph-janusgraph", "--no-default-features"]


[tasks.build-neo4j]
install_crate = { crate_name = "cargo-component", version = "0.20.0" }
command = "cargo-component"
args = ["build", "-p", "golem-graph-neo4j"]

[tasks.build-neo4j-portable]
install_crate = { crate_name = "cargo-component", version = "0.20.0" }
command = "cargo-component"
args = ["build", "-p", "golem-graph-neo4j", "--no-default-features"]


[tasks.release-build-arangodb]
install_crate = { crate_name = "cargo-component", version = "0.20.0" }
command = "cargo-component"
args = ["build", "-p", "golem-graph-arangodb", "--release"]

[tasks.release-build-arangodb-portable]
install_crate = { crate_name = "cargo-component", version = "0.20.0" }
command = "cargo-component"
args = ["build", "-p", "golem-graph-arangodb", "--release", "--no-default-features"]


[tasks.release-build-janusgraph]
install_crate = { crate_name = "cargo-component", version = "0.20.0" }
command = "cargo-component"
args = ["build", "-p", "golem-graph-janusgraph", "--release"]

[tasks.release-build-janusgraph-portable]
install_crate = { crate_name = "cargo-component", version = "0.20.0" }
command = "cargo-component"
args = [
"build",
"-p",
"golem-graph-janusgraph",
"--release",
"--no-default-features",
]


[tasks.release-build-neo4j]
install_crate = { crate_name = "cargo-component", version = "0.20.0" }
command = "cargo-component"
args = ["build", "-p", "golem-graph-neo4j", "--release"]

[tasks.release-build-neo4j-portable]
install_crate = { crate_name = "cargo-component", version = "0.20.0" }
command = "cargo-component"
args = ["build", "-p", "golem-graph-neo4j", "--release", "--no-default-features"]

[tasks.wit-update]
install_crate = { crate_name = "wit-deps-cli" }
command = "wit-deps"
args = ["update"]

[tasks.wit]
dependencies = ["wit-update"]

script_runner = "@duckscript"
script = """
modules = array graph arangodb janusgraph neo4j

for module in ${modules}
rm -r ${module}/wit/deps
mkdir ${module}/wit/deps/golem-graph
cp wit/golem-graph.wit ${module}/wit/deps/golem-graph/golem-graph.wit
cp wit/deps/wasi:io ${module}/wit/deps

echo "Copied WIT for module graph::${module}"
end

# Copy WIT files for integration tests for graph
rm -r ../test-graph/wit
mkdir ../test-graph/wit/deps/golem-graph
mkdir ../test-graph/wit/deps/io
cp wit/golem-graph.wit ../test-graph/wit/deps/golem-graph/golem-graph.wit
cp wit/deps/wasi:io/error.wit ../test-graph/wit/deps/io/error.wit
cp wit/deps/wasi:io/poll.wit ../test-graph/wit/deps/io/poll.wit
cp wit/deps/wasi:io/streams.wit ../test-graph/wit/deps/io/streams.wit
cp wit/deps/wasi:io/world.wit ../test-graph/wit/deps/io/world.wit

echo "Copied WIT for module test-graph"
"""

[tasks.build-test-components]
dependencies = ["build"]
install_crate = "cargo-binstall"
description = "Builds graph test components with golem-cli"
script = '''
cargo-binstall golem-cli@1.2.2-dev.11 --locked --no-confirm
cargo-binstall wac-cli --locked --no-confirm
cd ../test-graph

golem-cli --version
golem-cli app clean
golem-cli app build -b arangodb-debug
golem-cli app clean
golem-cli app build -b janusgraph-debug
golem-cli app clean
golem-cli app build -b neo4j-debug
'''
Loading
Loading