QuantNets

Convolutional Neural Networks (CNNs) vs. Quantized Graph Convolutional Networks (QGCNs) vs. Quantized Residual Graph Networks (QGRNs)

Generalizing CNNs to Graphs with Learnable Neighborhood Quantization

Dependencies

• PyTorch >= 1.1
• PyTorch geometric >= 1.1.2
• wget
• torch-geometric
• torch-cluster
• torch-sparse
• torch-scatter *** Install the dependencies via pip e.g.,:

pip install wget torch-geometric torch-cluster torch-sparse torch-scatter

Project Structure

In the root of the folder, there are three groups of directories:

1. Experiments (such as graph_classification, node_classification, eeg_autoencoder etc.) which contains scripts for generating empirical results
2. Libraries (including cnn, qgrn, qgcn, sgcn etc.) which contains the models and neural architectures, referenced by 'Experiment' scripts
3. Utils (utility directories like util, flops_counter) which contain helper functions used by the scripts
   • flops_counter - a third party library used for computing model complexity (library was modified to support our custom graph networks)
4. Custom Data (i.e., custom_data) - custom_data directory has instructions for generating benchmark graph datasets used in the paper
5. Extraneous (i.e., cache etc.) - cache holds all other scripts, google colab notebook templates etc.

Experiments: The core contents (leaf nodes) in these directories are either python notebooks or runnable scripts: in the latter, the folder is organized as shown below:

• _datasets.yaml (contains the datasets configuration for a given experiment - some of these configs are model specific parameters which are defined to ensure iso-model parameters across varying models)
• data.splits.yaml (contains the train-test splits for the experiment; you can refer to the datasets.yaml file's download_url key for a list of supported train-test splits for any given dataset )
• run.settings.yaml (contains all other configurable settings for the run, such as epochs[name: epochs], learning rate [name: lrs], how many times experiments should be repeated [name: runs] etc.)
• run_experiment*.py (primary script to be run to generate empirical results included in the paper)

Standard Datasets

To evaluate on full standard image dataset (i.e., MNIST, FashionMNIST, CIFAR10), please run the below to first download and compile a pytorch geometric dataset before running the corresponding python run_experiment*.py files (to train your models):

python3 ./graph_classification/generate_rawgraph_data.py -d MNIST

python3 ./graph_classification/generate_rawgraph_data.py -d FashionMNIST

python3 ./graph_classification/generate_rawgraph_data.py -d CIFAR10

NOTE:

• All empirical results generated will be populated into ./Experiments directory
• Adding the flag --profilemodels to run_experiment*.py's invocation will print out the model parameters, flops and wall clock times etc.
• Adding the flag --notraineval to run_experiment*.py's invocation will turn off evaluating training accuracies on the models
• As noted already, all notebooks and or scripts for generating empirical results are organized into the Experiments group of directories
• --dataset=standard will match all standard image datasets (i.e., MNIST, FashionMNIST, CIFAR10)
• --dataset=custom will match all custom datasets (i.e., datasets we post-processed from TUDatasets + our custom FEM Navier Stokes Datasets)
• --dataset=all will match all datasets (i.e., both custom and standard)

Experiments. Output YAML file Parsing:

• For each model trained per learning rate (across different runs of the same learning rate), results are aggregated into:
  • lr - learning rate
  • max - absolute max accuracy/loss over all epochs of model train and test across different runs per lr
  • smax - smoothened max where we average out the model's accuracy/loss over the last 5% of epochs per run
    • test_acc_avg_of_maxs: mean of the maxs across runs per learning rate of test data accuracy
    • test_acc_avg_of_smaxs: mean of the smaxs across runs per learning rate of test data accuracy
    • test_acc_max_of_maxs: max of the maxs across runs per learning rate of test data accuracy
    • test_acc_max_of_smaxs: max of the smaxs across runs per learning rate of test data accuracy
    • test_acc_std_of_maxs: standard deviation of the maxs across runs per learning rate of test data accuracy
    • test_acc_std_of_smaxs: standard deviation of the smaxs across runs per learning rate of test data accuracy
    • train_acc_avg_of_maxs: mean of the maxs across runs per learning rate of train data accuracy
    • train_acc_avg_of_smaxs: mean of the smaxs across runs per learning rate of train data accuracy
    • train_acc_max_of_maxs: max of the maxs across runs per learning rate of train data accuracy
    • train_acc_max_of_smaxs: max of the smaxs across runs per learning rate of train data accuracy
    • train_acc_std_of_maxs: standard deviation of the maxs across runs per learning rate of train data accuracy
    • train_acc_std_of_smaxs: standard deviation of the smaxs across runs per learning rate of train data accuracy
    • train_loss_avg_of_maxs: mean of the maxs across runs per learning rate of train data loss
    • train_loss_avg_of_smaxs: mean of the smaxs across runs per learning rate of train data loss
    • train_loss_max_of_maxs: max of the maxs across runs per learning rate of train data loss
    • train_loss_max_of_smaxs: max of the smaxs across runs per learning rate of train data loss
    • train_loss_std_of_maxs: standard deviation of the maxs across runs per learning rate of train data loss
    • train_loss_std_of_smaxs: standard deviation of the smaxs across runs per learning rate of train data loss
• As is shown in ./util/data_processing.py from the read_cached_graph_dataset() the data struct looks like below:
  • {
    • "raw": {
      • "x_train_data": ,
      • "y_train_data": ,
      • "x_test_data" : ,
      • "y_test_data" :
    • },
    • "geometric": {
      • "qgcn_train_data": ,
      • "qgcn_test_data" : ,
      • "sgcn_train_data": ,
      • "sgcn_test_data" :
    • }
  • }
  • NOTE: raw data is image data for training CNN models
    • also qgcn / sgcn are just placeholder names for graph datasets hence
    • you'll find in the codebase that we read say sgcn/qgcn data to train qgrn model etc.
    • NOTE: graph datasets (prefixed with either qgcn or sgcn) are not model-specific throughout this codebase & qgcn datasets == sgcn datasets

NOTES

1. Datasets, if not present, will be pulled in from dropbox / AWS wherever applicable
2. Results published in paper aggregated results from the stats below:
   • test_acc_avg_of_smaxs
   • test_acc_std_of_smaxs
   • train_acc_avg_of_smaxs
   • train_acc_std_of_smaxs
   • train_loss_avg_of_smaxs
   • train_loss_std_of_smaxs
3. Assuming early stopping / check-pointing of models, it'd be reasonable to have reported the max stats instead but we chose not to as it doesn't capture the dynamics during training as well as a smoothened (averaging) method does
4. Notice in the experimental runs that max stats also supports the same equivalence claim, and somewhat much more strongly than the average stats but due to reason 2. we decided to use average stats