X-Fi: A Modality-Invariant Foundation Model for Multimodal Human Sensing

This repository is the official code implementation of the paper X-Fi: A Modality-Invariant Foundation Model for Multimodal Human Sensing published on ICLR 2025.
The paper proposes the first foundation model that achieves modality-invariant multimodal human sensing.

🤖Authors

• Xinyan Chen, Jianfei Yang
• MARS Lab, School of Mechanical and Aerospace Engineering, Nanyang Technological University

⛷️Introduction

We introduce X-Fi, the first foundation model that achieves modality-invariant multimodal human sensing. This model would require training only once, allowing all sensor modalities that participated in the training process to be utilized independently or in any combination for a wide range of potential applications.

We evaluated X-Fi on HPE and HAR tasks in MM-Fi [1] and XRF55 [2], demonstrating that X-Fi surpasses previous methods by MPJPE 24.8% and PA-MPJPE 21.4% on the HPE task, and accuracy 2.8% on the HAR task.

⚙️Requirements

1. Install pytorch and torchvision (we use pytorch==2.1.1 and torchvision==0.16.1).
2. pip install -r requirements.txt

🧾Prepare Datasets and PT Model Weights

🎄🪚Download Processed Data

• Please download MM-Fi datatset and XRF55 datatset from their official websites.
• Remember the dataset saving dir for data loading process.
• Suggest to oragnize the downloaed dataset in the following structure:

X-Fi
├── Data
    ├── MMFi_Dataset
    ├── XRF55_Dataset

🪄✨Download Pretrained Modality-Specific Backbones or Pretrained X-Fi Model

• Please download Modality-Specific Backbones & Pretrained X-Fi Model from cloud storage.

For pretrained modality-specific backbones

Oragnize the downloaed .pt files into modality-corresponded sub-folders within each tasks's backbones or backbone_models folder.

Take the example of MMFi_HAR task, the organized structure will be:

X-Fi
├── MMFi_HAR
|   ├── backbones
|   |   ├── depth_benchmark
|   |   |   ├── depth_Resnet18.pt
|   |   ├── lidar_benchmark
|   |   |   ├── lidar_all_random.pt
|   |   ├── mmwave_benchmark
|   |   |   ├── mmwave_all_random_TD.pt
|   |   ├── RGB_benchmark
|   |   |   ├── RGB_Resnet18.pt

For pretrained X-Fi model

Unzip the downloaded pre-trained_weights folder into corresponding task main folder. e.g.

X-Fi
├── MMFi_HAR
|   ├── pre-trained_weights
|   |   ├── mmfi_har_checkpoint.pt

🏃‍♂️Run

Change Directory

Before run the scripts, cd into different task main folder directory.

Each task main folder is included in X-FI folder as follows:

X-Fi
├── MMFi_HAR
├── MMFi_HPE
├── XRF55_HAR

X-Fi Model Training

To train X-Fi model with default setting:

Run:

python run.py --dataset [path/to/corresponding/dataset]

Example:

<root_path>/X-Fi/MMFi_HAR > python run.py --dataset d:/Data/My_MMFi_Data/MMFi_Dataset

X-Fi Model Validation

To validate the trained X-Fi model performance on all modality combinations:

Run:

python validate_all.py --dataset [path/to/corresponding/dataset] --pt_weights [path/to/saved/pretrained/model/weights]

Example:

<root_path>/X-Fi/MMFi_HAR > python validate_all.py --dataset d:/Data/My_MMFi_Data/MMFi_Dataset --pt_weights ./pre-trained_weights/mmfi_har_checkpoint.pt

❤️‍🔥Citation

@inproceedings{chen2024xfi,
    title={X-Fi: A Modality-Invariant Foundation Model for Multimodal Human Sensing}, 
    author={Chen, Xinyan and Yang, Jianfei},
    booktitle = {International Conference on Learning Representations},
    Month = {April},
    year = {2025}
}