Merge branch 'main' into text_search_relations

barakb · web-flow · commit 81e45b2fd2ed · 2025-02-06T10:37:51.000+02:00
diff --git a/.wordlist.txt b/.wordlist.txt
@@ -342,3 +342,16 @@ thpool
 sds
 CRoaring
 RSALv
+
+hostnames
+bigmac
+calmcode
+io
+kafka
+readme
+github
+pre
+html
+body
+table
+Explainer
diff --git a/configuration.md b/configuration.md
@@ -84,6 +84,7 @@ The following table summarizes which configuration parameters can be set at modu
 | [EFFECTS_THRESHOLD](#effects_threshold)                      | V     | V     |
 | [CMD_INFO](#cmd_info)                                        | V     | V     |
 | [MAX_INFO_QUERIES](#max_info_queries)                        | V     | V     |
+| [IMPORT_FOLDER](#import_folder)                              | V     | X     |
 
 ---
 
@@ -379,3 +380,12 @@ total execution time / number of changes:  5ms / 5 = 1ms.
 if the average modification time is greater then `EFFECTS_THRESHOLD` the query
 will be replicated to both replicas and AOF as a graph effect otherwise the original
 query will be replicated.
+
+---
+
+### IMPORT_FOLDER
+
+The import folder configuration specifies an absolute path to a folder from which
+FalkorDB is allowed to load CSV files.
+
+Defaults to: `/var/lib/FalkorDB/import/`
diff --git a/cypher/load_csv.md b/cypher/load_csv.md
@@ -8,12 +8,12 @@ parent: "Cypher Language"
 
 # LOAD CSV
 
-```sh
+```cypher
 LOAD CSV FROM 'file://actors.csv' AS row
 MERGE (a:Actor {name: row[0]})
 ```
 
-`LOAD CSV FROM` accepts a string containing the path to a CSV file,
+`LOAD CSV FROM` accepts a string path to a CSV file,
 the file is parsed line by line, the current line is accessible through the 
 variable specified by `AS`. Each parsed value is treated as a `string`, use
 the right conversion functions e.g. `toInteger` to cast a value to its
@@ -25,9 +25,9 @@ Additional clauses can follow and accesses the `row` variable
 
 ### Importing local files
 
-FalkorDB defines a data directory ![see configuration](../configuration)
-Under which local CSV files should be stored, all `file://` URIs are resolved
-relatively to that directory.
+FalkorDB defines a data directory [see configuration](../configuration#import_folder)
+Under which local CSV files should be stored. All `file://` URIs are resolved
+relative to that directory.
 
 In the following example we'll load the `actors.csv` file into FalkorDB.
 
@@ -40,7 +40,7 @@ In the following example we'll load the `actors.csv` file into FalkorDB.
 | Chris Pratt    | 1979      |
 | Zoe Saldana    | 1978      |
 
-```sh
+```cypher
 LOAD CSV FROM 'file://actors.csv'
 AS row
 MERGE (a:Actor {name: row[0], birth_year: toInteger(row[1])})
@@ -63,7 +63,7 @@ In case the CSV contains a header row e.g.
 
 Then we should use the `WITH HEADERS` variation of the `LOAD CSV` clause
 
-```
+```cypher
 LOAD CSV WITH HEADERS FROM 'file://actors.csv'
 AS row
 MERGE (a:Actor {name: row[name], birth_year: toInteger(row[birthyear])})
@@ -95,15 +95,15 @@ We'll create a new graph connecting actors to the movies they've acted in
 
 Load actors:
 
-```sh
+```cypher
 LOAD CSV WITH HEADER FROM 'file://actors.csv'
 AS row
 MERGE (a:Actor {name:row['name']})
 ```
 
 Load movies and create `ACTED_IN` relations:
 
-```sh
+```cypher
 LOAD CSV WITH HEADER FROM 'file://acted_in.csv'
 AS row
 
@@ -112,3 +112,83 @@ MERGE (m:Movie {title: row['movie']})
 MERGE (a)-[:ACTED_IN]->(m)
 ```
 
+### Importing remote files
+
+FalkorDB supports importing remote CSVs via HTTPS.
+Here's an example loading the bigmac data-set from calmcode.io:
+
+```cypher
+LOAD CSV WITH HEADERS FROM 'https://calmcode.io/static/data/bigmac.csv' AS row
+RETURN row LIMIT 4
+
+1) 1) "ROW"
+2) 1) 1) "{date: 2002-04-01, currency_code: PHP, name: Philippines, local_price: 65.0, dollar_ex: 51.0, dollar_price: 1.27450980392157}"
+   2) 1) "{date: 2002-04-01, currency_code: PEN, name: Peru, local_price: 8.5, dollar_ex: 3.43, dollar_price: 2.47813411078717}"
+   3) 1) "{date: 2002-04-01, currency_code: NZD, name: New Zealand, local_price: 3.6, dollar_ex: 2.24, dollar_price: 1.60714285714286}"
+   4) 1) "{date: 2002-04-01, currency_code: NOK, name: Norway, local_price: 35.0, dollar_ex: 8.56, dollar_price: 4.088785046728971}"
+```
+
+### Dealing with a large number of columns or missing entries
+
+Loading data from CSV files that miss entries may cause complications.
+We've solved this (and made it useful for cases involving loading a large number of columns)
+with the following approach:
+
+Assuming this is the CSV file we're loading:
+
+
+### missing_entries.csv
+
+| name           | birthyear |
+| :--------------| :---------|
+| Lee Pace       | 1979      |
+| Vin Diesel     |           |
+| Chris Pratt    |           |
+| Zoe Saldana    | 1978      |
+
+>Note: both Vin Diesel and Chris Pratt are missing their birthyear entry
+
+When creating Actor nodes, there is no need to explicitly define each column as done previously.
+The following query creates an empty Actor node and assigns the current CSV row to it.
+This process automatically sets the node's attribute set to match the values of the current row:
+
+```cypher
+LOAD CSV FROM 'file://missing_entries.csv' AS row
+CREATE (a:Actor)
+SET a = row
+RETURN a
+
+1) 1) "a"
+2) 1) 1) 1) 1) "id"
+            2) (integer) 0
+         2) 1) "labels"
+            2) 1) "Actor"
+         3) 1) "properties"
+            2) 1) 1) "name"
+                  2) "Zoe Saldana"
+               2) 1) "birthyear"
+                  2) "1978"
+   2) 1) 1) 1) "id"
+            2) (integer) 1
+         2) 1) "labels"
+            2) 1) "Actor"
+         3) 1) "properties"
+            2) 1) 1) "name"
+                  2) "Chris Pratt"
+   3) 1) 1) 1) "id"
+            2) (integer) 2
+         2) 1) "labels"
+            2) 1) "Actor"
+         3) 1) "properties"
+            2) 1) 1) "name"
+                  2) "Vin Diesel"
+   4) 1) 1) 1) "id"
+            2) (integer) 3
+         2) 1) "labels"
+            2) 1) "Actor"
+         3) 1) "properties"
+            2) 1) 1) "name"
+                  2) "Lee Pace"
+               2) 1) "birthyear"
+                  2) "1979"
+```
diff --git a/datatypes.md b/datatypes.md
@@ -136,6 +136,19 @@ $ redis-cli GRAPH.QUERY G "MATCH (n) RETURN n {.name, .age} AS projection"
 2) 1) 1) "{name: Jeff, age: 32}"
 ```
 
+#### Map merging
+
+You can combine two maps, where values in the second map will override corresponding values in the first map.
+For example:
+
+```sh
+$ redis-cli GRAPH.QUERY g "RETURN {a: 1, b: 2} + {a: 2, c: 3}"
+1) 1) "{a: 1, b: 2} + {a: 2, c: 3}"
+2) 1) 1) "{b: 2, a: 2, c: 3}"
+3) 1) "Cached execution: 0"
+   2) "Query internal execution time: 0.467666 milliseconds"
+```
+
 #### Function calls in map values
 
 The values in maps and map projections are flexible, and can generally refer either to constants or computed values:
diff --git a/index.md b/index.md
@@ -13,8 +13,7 @@ permalink: /
 
 [![Try Free](https://img.shields.io/badge/Try%20Free-FalkorDB%20Cloud-FF8101?labelColor=FDE900&style=for-the-badge&link=https://app.falkordb.cloud)](https://app.falkordb.cloud)
 
-
-FalkorDB is a blazing fast graph database used for low latency & high throughput scenarios, under the hood it runs [GraphBLAS](http://faculty.cse.tamu.edu/davis/GraphBLAS.html)  to perform graph operations using sparse linear algebra.
+FalkorDB is a blazing fast graph database used for low latency & high throughput scenarios. Under the hood, it runs [GraphBLAS](http://faculty.cse.tamu.edu/davis/GraphBLAS.html) to perform graph operations using sparse linear algebra.
 
 ## Primary features
 
@@ -23,7 +22,7 @@ FalkorDB is a blazing fast graph database used for low latency & high throughput
 * Offers [Full-Text Search](/cypher/indexing#full-text-indexing), [Vector Similarly](/cypher/indexing#vector-indexing) & [Numeric indexing](/cypher/indexing).
 * Interacts via either [RESP](https://redis.io/docs/reference/protocol-spec/) and [Bolt](https://en.wikipedia.org/wiki/Bolt_(network_protocol)) protocols
 * Graphs represented as sparse adjacency matrices
-
+* Supports GraphRAG with the [GraphRAG SDK](https://github.com/FalkorDB/GraphRAG-SDK) for advanced graph reasoning and generative AI tasks.
 
 ## Give it a try
 
diff --git a/integration/index.md b/integration/index.md
@@ -14,4 +14,6 @@ Learn how to leverage FalkorDB's flexible APIs and SDKs to build high-performanc
 ## Topics in This Section
 
 - [REST API](./rest.md): Learn how to interact with FalkorDB using its REST API for seamless integration with your applications.
+- [Kafka Connect](./kafka-connect.md): Learn how to interact with FalkorDB using Kafka Connect sink to replicate data from third-party applications.
+
 
diff --git a/integration/kafka-connect.md b/integration/kafka-connect.md
@@ -0,0 +1,108 @@
+---
+title: "Kafka Connect Sink"
+nav_order: 2
+description: "Kafka Connect sink detailed doc"
+parent: "integration"
+---
+
+![FalkorDB x Kafka Connect Banner](https://github.com/user-attachments/assets/941bb532-8613-4135-b4c9-232a700da314)
+
+
+
+## Get Started
+
+- [Obtaining the connector](#obtaining-the-connector)
+- [Configuring the connector](#configuring-the-connector)
+- [Kafka message format](#kafka-message-format)
+
+---
+
+### **1️⃣ Obtaining the Connector**
+
+You can build the connector from [source](https://github.com/FalkorDB/falkordb-kafka-connect) or download the pre-built  [JAR](https://github.com/FalkorDB/falkordb-kafka-connect/releases/download/v1.0.0/falkordb-kafka-connect-uber.jar) file from the releases. The GitHub repository includes a README with instructions for running the connector locally. The [GitHub](https://github.com/FalkorDB/falkordb-kafka-connect?tab=readme-ov-file#how-to-run-the-example) repository includes a README with instructions for running the connector locally.
+
+### **2️⃣ Configuring the Connector**
+
+Kafka Connector Properties Overview: 
+This document explains the properties required to configure the FalkorDB Sink Connector for Apache Kafka. 
+>Configurations should be specified in a properties file format.
+
+#### Properties Overview
+
+| **Property**                  | **Description**                                                                                                                                  |
+|--------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------|
+| `name`                        | Specifies the unique name of the connector instance, e.g., `falkordb-connector`. This name identifies the connector in the Kafka Connect framework. |
+| `connector.class`             | Defines the Java class that implements the connector logic. Use `com.falkordb.FalkorDBSinkConnector` to write data from Kafka topics to FalkorDB. |
+| `tasks.max`                   | Sets the maximum number of tasks for the connector. A value of `1` uses a single task. Increasing this can boost throughput but requires resources. |
+| `topics`                      | Specifies the Kafka topic(s) to consume messages from. Set to `falkordb-topic` to read messages from this topic.                                  |
+| `key.converter`               | Defines the converter class for message keys. `StringConverter` treats keys as simple strings.                                                   |
+| `value.converter`             | Specifies the converter for message values. `StringConverter` treats values as strings.                                                           |
+| `value.converter.schemas.enable` | Indicates whether schemas should be included with message values. Setting to `false` excludes schema information.                                |
+| `falkor.url`                  | Specifies the connection URL for FalkorDB. Example: `redis://localhost:6379`. Essential for connecting Kafka to FalkorDB.                         |
+
+
+
+>The above properties configure a Kafka Sink Connector that reads messages from a specified topic and writes them into
+FalkorDB using string conversion for both keys and values. Adjusting these properties allows you to tailor the
+connector's behavior according to your application's requirements.
+
+
+## Configuration Example
+
+  ```properties
+name=falkordb-connector
+connector.class=com.falkordb.FalkorDBSinkConnector
+tasks.max=1
+topics=falkordb-topic
+key.converter=org.apache.kafka.connect.storage.StringConverter
+value.converter=org.apache.kafka.connect.storage.StringConverter
+value.converter.schemas.enable=false
+falkor.url=redis://localhost:6379
+  ```
+
+## Kafka Message Format
+
+#### JSON Structure Overview
+
+The message is an array containing multiple objects, each representing a command to be executed on the graph database.
+Below is a breakdown of the key components of each message object.
+
+Example:
+
+```json
+[
+  {
+    "graphName": "falkordb",
+    "command": "GRAPH_QUERY",
+    "cypherCommand": "CREATE (p:Person {name: $name_param, age: $age_param, location: $location_param}) RETURN p",
+    "parameters": {
+      "location_param": "Location 0",
+      "age_param": 20,
+      "name_param": "Person 0"
+    }
+  },
+  {
+    "graphName": "falkordb",
+    "command": "GRAPH_QUERY",
+    "cypherCommand": "CREATE (p:Person {name: $name_param, age: $age_param, location: $location_param}) RETURN p",
+    "parameters": {
+      "location_param": "Location 1",
+      "age_param": 21,
+      "name_param": "Person 1"
+    }
+  }
+]
+
+```
+
+#### Key Components
+
+The table below explains essential properties for executing commands in FalkorDB through Kafka messages.
+
+| **Property**      | **Description**                                                                                                           | **Example**                                                                                             | **Explainer**                                                                                         |
+|-------------------|-----------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------|
+| `graphName`       | Specifies the name of the graph database where the command will be executed.                                              | `"falkordb"`. Kafka messages can update multiple graphs.                                                |                                                    |
+| `command`         | Indicates the type of operation being performed. `"GRAPH_QUERY"` means a query will be executed against the graph database. | `"GRAPH_QUERY"`                                                                                          |                                                   |
+| `cypherCommand`   | Contains the actual Cypher query to be executed. Cypher is a query language for graph databases.                           | ```cypher CREATE (p:Person {name: $name_param, age: $age_param, location: $location_param}) RETURN p ``` | Creates a `Person` node with `name`, `age`, and `location` properties.                                |
+| `parameters`      | Holds key-value pairs for placeholders in the `cypherCommand`.                                                             | ```json {"name_param": "Person 0", "age_param": 20, "location_param": "Location 0"} ```                  | Used to define properties for the new node.                                                           |
+
diff --git a/operations/cluster.md b/operations/cluster.md
diff --git a/operations/k8s_support.md b/operations/k8s_support.md
