Update READMEs

IlyaMuravjov · IlyaMuravjov · commit 897235d47cbb · 2024-03-12T23:12:06.000+03:00
diff --git a/README.md b/README.md
@@ -1,9 +1,12 @@
-[![CircleCI](https://circleci.com/gh/JetBrains-Research/CFPQ_PyAlgo/tree/master.svg?style=svg)](https://circleci.com/gh/JetBrains-Research/CFPQ_PyAlgo/tree/master)
+## CFPQ_PyAlgo
 
-# CFPQ_PyAlgo
-The CFPQ_PyAlgo is a repository for developing, testing and benchmarking algorithms that solve Formal-Language-Constrained Path Problems, such as Context-Free Path Queries and Regular Path Queries. All algorithms are based on the [GraphBLAS](http://graphblas.org/index.php?title=Graph_BLAS_Forum) framework that allows you to represent graphs as matrices and work with them in terms of linear algebra. For convenience, all the code is written in Python using [pygraphblas](https://github.com/michelp/pygraphblas) or in C/C++ using purely [SuiteSparse](https://github.com/DrTimothyAldenDavis/SuiteSparse/tree/master/GraphBLAS) with a Python wrapper. 
+The CFPQ_PyAlgo is a repository for developing, testing and evaluating solvers for
+Formal-Language-Constrained Path Problems, such as Context-Free Path Queries and Regular Path Queries.
 
-# Installation
+All algorithms are based on the [GraphBLAS](http://graphblas.org/index.php?title=Graph_BLAS_Forum) framework that allows to represent graphs as matrices
+and work with them in terms of linear algebra.
+
+## Installation
 First of all you need to clone repository with its submodules:
 
 ```bash
@@ -12,26 +15,27 @@ cd CFPQ_PyAlgo/
 git submodule init
 git submodule update
 ```
-Then the easiest way to get started is to use Docker. An alternative, which is more correct, is to install everything directly.
+Then the easiest way to get started is to use Docker. An alternative is to install everything directly.
 
-## Using Docker
+### Using Docker
 The first way to start is to use Docker:
 
 ```bash
 # build docker image
-docker build --tag <some_tag> .
+docker build --tag cfpq_py_algo .
 
 # run docker container
-docker run --rm -it -v ${PWD}:/CFPQ_PyAlgo <some_tag> bash
+docker run --rm -it -v ${PWD}:/CFPQ_PyAlgo cfpq_py_algo bash
 ```
-After it you can develop everything locally and run tests and benchmarks inside the container. Also you can use PyCharm and [configure an interpreter using Docker]( https://www.jetbrains.com/help/pycharm/using-docker-as-a-remote-interpreter.html).
+After it, you can develop everything locally and run tests and benchmarks inside the container. 
+Also, you can use PyCharm Professional and [configure an interpreter using Docker](https://www.jetbrains.com/help/pycharm/using-docker-as-a-remote-interpreter.html).
 
-## Direct install
-The correct way is to install everything into your local python interpreter or virtual environment.
+### Direct install
+The other way is to install everything into your local python interpreter or virtual environment.
 
 First of all you need to install [pygraphblas](https://github.com/michelp/pygraphblas) package.
 ```bash
-pip3 install pygraphblas
+pip3 install pygraphblas==5.1.8.0
 ```
 Secondly you need to install cfpq_data_devtools package and other requirements:
 
@@ -47,56 +51,32 @@ To check if the installation was successful you can run simple tests
 ```bash
 python3 -m pytest test -v -m "CI"
 ```
-# Benchmark
-Look please [Readme](https://github.com/JetBrains-Research/CFPQ_PyAlgo/blob/master/benchmark/README.md) in *benchmark*
-
-# Usage
-
-Let's describe an example of using the implementation outside this environment.
 
-For example, you want to solve a basic problem CFPQ using the matrix algorithm. To do this, you need a context-free grammar (**Gr**), as well as a graph (**G**) in the format of "triplets". 
+## CLI
+CFPQ_Algo provides a command line interface for running 
+all-pairs CFPQ solver with relation query semantics.
 
-Then the matrix algorithm can be run as follows, where *PATH_TO_GRAMMAR* --- path to file with **Gr**, *PATH_TO_GRAPH* --- path to file with **G**
+See [cfpq_cli/README](cfpq_cli/README.md) for more details.
 
-```cython
-from src.problems.Base.algo.matrix_base.matrix_base import MatrixBaseAlgo
-from cfpq_data import cfg_from_txt
-from src.graph.graph import Graph
+## Evaluation
 
-from pathlib import Path
+CFPQ_PyAlgo provides scripts for performing evaluating performance 
+of various CFPQ solvers (icluding third-party ones).
 
-algo = MatrixBaseAlgo()
-algo.prepare(Graph.from_txt(Path(PATH_TO_GRAPH)), cfg_from_txt(Path(PATH_TO_GRAMMAR)))
-res = algo.solve()
-print(res.matrix_S.nvals)
-```
-The given fragment displays the number of pairs of vertices between which the desired path exists.
-
-More examples can be found in *test*
+See [cfpq_eval/README](cfpq_eval/README.md) for more details.
 
-# Project structure
+## Project structure
 The global project structure is the following:
 
 ```
-.
+├── cfpq_algo - new optimized CFPQ algorithm implementations
+├── cfpq_cli - scripts for running CFPQ algorithms
+├── cfpq_eval - scripts for evaluating performance of various CFPQ solvers (icluding third-party ones)
+├── cfpq_matrix - matrix wrappers that improve performance of operations with matrices
+├── cfpq_model - graph & grammar representations
 ├── deps
 │   └── CFPQ_Data - repository with graphs and grammars suites
-├───benchmark - directory for performance measurements of implementations
-├── src
-│   ├── problems - directory where all the problems CFPQ that we know how to solve
-│   │   ├───AllPaths
-│   │   ├───Base
-│   │   ├───MultipleSource
-│   │   └───SinglePath
-│   ├── grammar - directory for all grammar formats representation and its loading  
-│   ├── graph - directory for all graph formats representation and its loading
-│   └── utils - directory for other useful classes and methods
-└── test
-    ├───AllPaths - tests for implementations in src.problems.AllPaths
-    ├───Base - tests for implementations in src.problems.Base
-    ├───data - dataset for tests
-    ├───MultipleSource - tests for implementations in src.problems.MultipleSource
-    ├───SinglePath - tests for implementations in src.problems.SinglePath
-    └───suites
-
+├── benchmark - directory for performance measurements of legacy CFPQ implementations
+├── src - legacy CFPQ implementations
+└── test - tests
 ```
diff --git a/benchmark/README.md b/benchmark/README.md
@@ -1,4 +1,14 @@
-# How to start
+# LEGACY WARNING
+
+This folder contains benchmarks for legacy CFPQ implementations.
+
+New optimized CFPQ implementations are being implemented in [cfpq_algo](../cfpq_algo) folder
+and can be evaluated with scripts in [cfpq_eval](../cfpq_eval) folder.
+
+<details>
+<summary>Full README</summary>
+
+## How to start
 First, create a directory for the dataset. It should have two subdirectories for graphs (Graphs) and grammars (Grammars). In the second step, select an algorithm for benchmarking. Then run the command: 
 ```
 python3 -m benchmark.start_benchmark.py -algo ALGO -data_dir DATA_DIR
@@ -10,8 +20,10 @@ There are also a number of optional parameters:
 + -result_dir --- specify a directory for uploading the results
 + -max_len_paths --- Limit on the length of the retrieved paths 
 
-# Add new algorithm
+## Add new algorithm
 To add a new implementation of the algorithm to the list of available measurements, you must:
 1. Add you algorithm in *algo_impl.ALGO_PROBLEM*
 2. Add you implementation in *algo_impl.ALGO_IMPL*
-3. Create new or use the existing pipeline or  in *bench.benchmark* 
+3. Create new or use the existing pipeline or  in *bench.benchmark*
+
+</details>
diff --git a/cfpq_cli/README.md b/cfpq_cli/README.md
@@ -0,0 +1,106 @@
+# CFPQ_CLI
+
+The `cfpq_cli` module provides a Command Line Interface (CLI) for solving
+Context-Free Language Reachability (CFL-r) problem for all vertex pairs 
+in a graph with respect to a specified context-free grammar.
+
+## Getting Started
+
+Ensure the CFPQ_PyAlgo project is properly set up on your system before using the CLI.
+Setup instructions are available in the project's main [README](../README.md).
+
+## Usage
+
+### Running the Script
+
+For detailed information on script options, execute the following command:
+
+```bash
+cd .. # Should be run from CFPQ_PyAlgo project root directory
+python3 -m cfpq_cli.run_all_pairs_cflr --help
+```
+
+The basic command usage is as follows:
+
+```
+python3 -m cfpq_cli.run_all_pairs_cflr [OPTIONS] ALGORITHM GRAPH GRAMMAR
+```
+
+- `ALGORITHM` selects the algorithm. The available options are `IncrementalAllPairsCFLReachabilityMatrix` and `NonIncrementalAllPairsCFLReachabilityMatrix`.
+- `GRAPH` specifies the path to the graph file.
+- `GRAMMAR` indicates the path to the grammar file.
+
+#### Optional Arguments
+
+- `--time-limit TIME_LIMIT` sets the maximum execution time in seconds.
+- `--out OUT` specifies the output file for saving vertex pairs.
+- `--disable-optimize-block-matrix` disables the optimization of block matrices.
+- `--disable-optimize-empty` disables the optimization for empty matrices.
+- `--disable-lazy-add` disables lazy addition optimization.
+- `--disable-optimize-format` disables optimization of matrix formats.
+
+### Example
+
+To solve the CFL-R problem using an incremental algorithm with a 60-second time limit for 
+[indexed_tree.g](../test/pocr_data/indexed_an_bn/indexed_tree.g) and 
+[an_bn_indexed.cnf](../test/pocr_data/indexed_an_bn/an_bn_indexed.cnf) and get results in 
+[results.txt](../results.txt) execute:
+
+```bash
+cd .. # Should be run from CFPQ_PyAlgo project root directory
+python3 -m cfpq_cli.run_all_pairs_cflr \
+    IncrementalAllPairsCFLReachabilityMatrix \
+    test/pocr_data/indexed_an_bn/indexed_tree.g \
+    test/pocr_data/indexed_an_bn/an_bn_indexed.cnf \
+    --time-limit 60 \
+    --out results.txt
+```
+
+### Grammar Format
+
+The grammar file should be formatted with each production rule on a separate line, adhering to the following schema:
+
+```
+<LEFT_SYMBOL>	[RIGHT_SYMBOL_1]	[RIGHT_SYMBOL_2]
+```
+
+- `<LEFT_SYMBOL>`: the symbol on the left-hand side of a production rule.
+- `<RIGHT_SYMBOL_1>` and `<RIGHT_SYMBOL_2>`: the symbols on the right-hand side of the production rule, each of them is optional.
+- The symbols must be separated by whitespace.
+- The last two line specify the start symbol in the format 
+  ```
+  Count:
+  <START_SYMBOL>
+  ```
+
+#### Example
+```
+S	AS_i	b_i
+AS_i	a_i	S
+S	c
+
+Count:
+S
+```
+
+### Graph Format
+
+The graph file should represent edges using the format:
+
+```
+<EDGE_SOURCE>	<EDGE_DESTINATION>	<EDGE_LABEL>	[LABEL_INDEX]
+```
+
+- `<EDGE_SOURCE>` and `<EDGE_DESTINATION>`: specify the source and destination nodes of an edge.
+- `<EDGE_LABEL>`: the label associated with the edge.
+- `[LABEL_INDEX]`: an optional index for labels with subscripts, indicating the subscript value.
+- The symbols must be separated by whitespace
+- Labels with subscripts must end with "\_i". For example, an edge $1 \xrightarrow{x_10} 2$ is denoted by `1	2	x_i	10`.
+
+#### Example
+```
+1	2	a_i	1
+2	3	b_i	1
+2	4	b_i	2
+1	5	c
+```
diff --git a/cfpq_eval/README.md b/cfpq_eval/README.md
@@ -0,0 +1,86 @@
+# CFPQ Evaluation
+
+The `cfpq_eval` module in CFPQ_PyAlgo evaluates performance of CFPQ solvers,
+integrating with both CFPQ_PyAlgo itself and third-party tools.
+
+## Setting up the environment
+
+Build and run a Docker container for evaluation using [Dockerfile-all-tools](../Dockerfile-all-tools).
+
+Build Docker image:
+```
+docker build -f Dockerfile-all-tools -t cfpq_eval .
+```
+
+Run Docker container:
+```
+docker run -it cfpq_eval bash
+```
+
+## Running the Script
+
+For detailed information on evaluation script options, execute the following command:
+
+```bash
+cd .. # Should be run from CFPQ_PyAlgo project root directory
+python3 -m cfpq_cli.run_all_pairs_cflr --help
+```
+
+The basic command usage is as follows:
+
+```
+python3 -m cfpq_eval.eval_all_pairs_cflr algo_config.csv data_config.csv results_path [--rounds ROUNDS] [--timeout TIMEOUT]
+```
+
+- `algo_config.csv` specifies algorithm configurations.
+- `data_config.csv` specifies the dataset.
+- `results_path` specifies path for saving raw results.
+- `--rounds` sets run times per config (default is 1).
+- `--timeout` limits each configuration's execution time in seconds (optional).
+
+## Configuration Files
+
+### Algorithm Configuration
+
+The `algo_config.csv` outlines algorithms and settings. Supported algorithms:
+
+- `IncrementalAllPairsCFLReachabilityMatrix`
+- `NonIncrementalAllPairsCFLReachabilityMatrix`
+- `pocr`
+- `pearl`
+- `graspan`
+- `gigascale`
+
+For Matrix-based algorithms options described in [cfpq_cli/README](../cfpq_cli/README.md).
+can be used to alter the behaviour.
+
+#### Example
+
+```
+algo_name,algo_settings
+"Matrix (some optimizations disabled)",IncrementalAllPairsCFLReachabilityMatrix --disable-optimize-empty --disable-lazy-add
+"pocr",pocr
+```
+
+### Data Configuration
+
+The `data_config.csv` pairs graph and grammar files, 
+referenced files should be in format described in [cfpq_cli/README](../cfpq_cli/README.md).
+
+#### Example
+
+```
+graph_path,grammar_path
+data/graphs/aa/leela.g,data/grammars/aa.cnf
+data/graphs/java/eclipse.g,data/grammars/java_points_to.cnf
+```
+
+## Interpreting Results
+
+Raw data is saved to `results_path`, while quick summary including mean execution time,
+memory usage, and output size are rendered in standard output stream.
+
+## Custom Tools Integration
+
+Additional CFPQ solvers can be supported to evaluation by implementing `AllPairsCflrToolRunner` interface
+and updating `run_appropriate_all_pairs_cflr_tool()` function.
diff --git a/src/README.md b/src/README.md
