NeuroKinect4K-MOI-SIM Dataset Extraction Code

This code is complementary to the NeuroKinect4K-MOI-SIM dataset, first presented in the paper NeuroKinect4K-MOI-SIM: A Simulated Yet Real Benchmark Dataset for Keypoint-Based Action, Gesture, and Emotion Recognition in Epileptic Seizure Semiology, allowing the extraction of all video and skeleton data for each Movement of Interest (MOI) independently. This code is only relevant for reorganize the dataset in a label-based way, instead of a subject-based one.

The 33 classes, their description and metrics of this dataset can be seen on the next table.

Directory Structure

The code generates a CSV file containing information about all present MOIs, specifically:

• subject - the code for the folder of origin, of structure S_{subject}_{repetition} (eg. S01_01, S02_01, etc.)
• label- the id of the MOI label (from 0 to 32)
• start_video - the name of the video where the MOI starts
• start_frame - the number of the frame, on the start_video, where the MOI starts
• end_video - the name of the video where the MOI ends
• end_frame - the number of the frame, on the end_video, where the MOI end

Each MOI is uniquely identified by {subject}_L{label}_{occurrence}. For example:

• The first time subject S01_01 performs an MOI with label 1, it will be identified as S01_01_L1_1.
• The second time the same subject performs the same MOI with label 1, it will be identified as S01_01_L1_2, and so on.

For each MOI, the corresponding video and skeleton data for both keypoint models (Sapiens 133 (2B parameters) and 308 (1B parameters)) are created.

Once you run this code, a directory structure will be created as follows:

Input layout (original structure of NeuroKinect4K-MOI-SIM)

- MOI_SIM/
    - S01_01/
        - 20230313_092157_369/
            - 20230313_092157_369_blurred.mp4
            - 20230313_092157_369_blurred_sapiens133kp.mp4
            - 20230313_092157_369_blurred_sapiens308kp.mp4
            - 20230313_092157_369.json
            - Sapiens_133kp_result
                    - frame_000001.json
                    - frame_000002.json
                    - ...
            - Sapiens_308kp_result
                    - frame_000001.json
                    - frame_000002.json
                    - ...  
        - ...
    - ...
    - S08_01/
      - ...

Example Output Layout:

/MOI_Sim/MOI_by_label/
└── 1/
    └── S01_01_L1_1/
        ├── S01_01_L1_1.mp4
        ├── Sapiens_133kp_result/
        │   ├── frame_000001.json
        │   ├── ...
        └── Sapiens_308kp_result/
            ├── frame_000001.json
            ├── ...

Requirements

To run this code, you need to have FFMPEG installed on your system. This code was tested with ffmpeg version 4.2.7-0ubuntu0.1.

FFMPEG Installation

You can install FFMPEG on Ubuntu using the following commands:

sudo apt update

sudo apt install ffmpeg

And check with:

ffmpeg -version

Additional Requirements:

• Python: The code is written in Python 3, and you'll need to have a Python environment ready.
• Necessary Python Libraries: Ensure the following libraries are installed:
  • numpy
  • csv
  • shutil
  • json
  • subprocess
  • os

You can install the necessary libraries using pip:

pip install numpy

Running the Code

To execute the code, you need to run the following command in your terminal:

Command Format

python MOIS_extractor.py \
  --base_folder /path/to/MOI_Simulation \
  --output_dir /path/to/output_clips \
  --subjects S01_01 S02_01 S03_01 \
  --csv_output extracted_annotations.csv

Parameters:

• --base_folder (Required): Full path to the root folder containing the subject/video data. This folder should include all the video and skeleton data for the subjects.

• --csv_output (Optional): Path to save the annotation CSV. This CSV will contain the extracted data, such as subject, label, start video, start frame, end video, and end frame. Default: annotations.csv

• --output_dir (Required): Full path to the directory where the extracted clips and keypoints will be saved. This directory will contain the video clips as well as the corresponding keypoints data.

• --subjects (Required): A list of subject folder names (e.g., S01_01 S02_01 S03_01). These are the subjects for which the data will be processed.

• --fps (Optional): The frame rate of the video. The default is 30 frames per second. Default: 30

Example Command

Here's a complete example of running the script with specific parameters:

python MOIS_extractor.py \
  --base_folder /MOI_Sim/raw_data \
  --output_dir /MOI_Sim/MOIs_by_label \
  --subjects S01_01 S02_01 S03_01 \
  --csv_output extracted_annotations.csv

Conclusion

This code allows the extraction of both video and skeleton data (for the Sapiens 133kp and Sapiens 308kp models) for each MOI. It organizes the extracted data into a directory structure that is easy to understand and use for further analysis.

Make sure to have FFMPEG installed and configure your paths and parameters correctly to start extracting the data successfully.

License

The code in this repository is available under a MIT license. It does, however, depend on code from third parties such as FFmpeg, so if you do use it make sure to comply with the licenses of the dependency.

Citations

If you use NeuroKinect4K-MOI-SIM, you agree to cite the correspondent paper:

@inproceedings{karacsony2025neurokinect4k,
 title     = {NeuroKinect4K-MOI-SIM: A Simulated Yet Real Benchmark Dataset for Keypoint-Based Action, Gesture, and Emotion Recognition in Epileptic Seizure Semiology},
 author    = {Karácsony, Tamás and Bessa, Mariana and Fearns, Nicholas and Rémi, Jan and Silva Cunha, João Paulo},
 booktitle = {Submitted},
 year      = {2025}
}

Support

For more information contact Mariana Bessa at mariana.bessa@inesctec.pt or Tamás Karacsony at tamas.karacsony@inesctec.pt

Authors and acknowledgment

The code was authored by Mariana Bessa under the guidance and supervision of Tamás Karacsony and João Paulo Cunha, all 3 of whom are affiliated with INESC TEC.