ParaGrapher: Graph Loading API and Library

https://blogs.qub.ac.uk/DIPSA/ParaGrapher

This repository contains the source code of ParaGrapher, an API and library for loading graphs. For futher information about the library please refer to https://blogs.qub.ac.uk/DIPSA/ParaGrapher/ and publications.

ParaGrapher API Documentation

Please visit the Wiki or download the PDF file using this link.

Implemented Graph Types in ParaGrapher Library

• PARAGRAPHER_CSX_WG_400_AP : WebGraphs with 4 Bytes ID per vertex without weights on edges or vertices
• PARAGRAPHER_CSX_WG_800_AP : Big WebGraphs with 8 Bytes ID per vertex without weights on edges or vertices
• PARAGRAPHER_CSX_WG_404_AP : WebGraphs with 4 Bytes ID per vertex and 4 Bytes integer weight per edge and without weights on vertices

Requirements

• gcc with a version greater than 9
• JDK with a version greater than 15
• bc, wget, and unzip
• libfuse3 and libnuma for using pg_fuse (optional)

Download Sample Datasets

• Run make download_WG400, make download_WG404, or make download_WG800
  to download and store sample datasets into the test/datasets folder.

Compiling and Executing Code

• By commenting -DNDEBUG in Line 19 of the Makefile, ParaGrapher will output its logs.

• With make all the C and Java source codes are compiled and the required WebGraph libraries are downloaded.

• All compiled and downloaded files are stored in the lib64 folder and future calls to the library requires setting the PARAGRAPHER_LIB_FOLDER environemnt variable to the lib64 folder.

• The test folder contains sample codes for different types of graphs. You may pass argument dataset to specify the location of the test, e.g., make test1_deg_dist_WG400 dataset=path/to/dataset.

• In the first access to the graphs in WebGraph format a delay may be experienced for creating two files by the library:

  1. A WebGraph .offset file is required which is created through a call to the WebGraph framework.
  2. An _offsets.bin file is created that contains the offsets array of the CSX format but in binary and littel-endian format with 8-Bytes values for each of |V|+1 elements. In case of MS-BioGraphs, the file with name MS??_offsets.bin can be downloaded and renamed as MS??-underlying_offsets.bin to prevent creating.

• ParaGrapher creates shared memory objects (in /dev/shm) with names starting by paragrapher_ for communication between C and Java sides. The files are deleted at the end of a successful exuection. Otherwise, they should be manually deleted using make clean-shm-files.

• After calling ParaGrapher, the cached contents of the storage should be dropped using echo 3 > /proc/sys/vm/drop_caches or by calling the flushcache program that has the same functionality but with a longer execution time.

ParaGrapher FUSE File System (PG-FUSE)

To accelerate the loading WebGraphs, ParaGrapher introduces pg_fuse, a custom file system built on top of the FUSE framework.

To enable pg_fuse, simply pass USE_PG_FUSE as an argument in the args parameter of the paragrapher_open_graph() function.

For more detailed information, refer to PG-FUSE documentation.

Bandwidth Measurement

• The file test/read_bandwidth.c contains a benchmark implemented in C to measure the read bandwidth of storage for (i) different thread numbers, (ii) different block sizes, and (iii) different read methods (read(), pread(), mmap()).

• The file test/ReadBandwidth.java contains a benchmark implemented in Java to measure the read bandwidth of storage for (i) different thread numbers, (ii) different block sizes, and (iii) different read methods (read(), mmap()). The script test/java-read-bandwidth.sh may be used for changing parameters.

• The Storage Bandwidth Evaluation shows the execution results of the above programs for three storage types, SSD, HDD, and LustreFS.

Citation

@misc{pg_fuse,
      title={Accelerating Loading WebGraphs in ParaGrapher}, 
      author={Mohsen {Koohi Esfahani}},
      year={2025},
      eprint={2507.00716},
      archivePrefix={arXiv},
      primaryClass={cs.DC},
      url={https://arxiv.org/abs/2507.00716}, 
}

@misc{paragrapher-arxiv,
  title = { Selective Parallel Loading of Large-Scale 
            Compressed Graphs with {ParaGrapher}}, 
  author = { {Mohsen} {Koohi Esfahani} and Marco D'Antonio and 
             Syed Ibtisam Tauhidi and Thai Son Mai and 
             Hans Vandierendonck},
  year = {2024},
  eprint = {2404.19735},
  archivePrefix = {arXiv},
  primaryClass = {cs.AR},
  doi = {10.48550/arXiv.2404.19735},
  url={https://arxiv.org/abs/2404.19735}, 
}

License

Licensed under the GNU v3 General Public License, as published by the Free Software Foundation. You must not use this Software except in compliance with the terms of the License. Unless required by applicable law or agreed upon in writing, this Software is distributed on an "as is" basis, without any warranty; without even the implied warranty of merchantability or fitness for a particular purpose, neither express nor implied. For details see terms of the License (see attached file: LICENSE).