Active Learning for Extended 2D Body Keypoint Dataset Creation

This project involves the development of a comprehensive 2D body keypoints dataset, created as an extension of existing COCO and MPII keypoints. The extended dataset includes additional keypoints to better represent the spinal cord, providing more detail for pose estimation tasks.

Our approach involves an initial automated generation of the extended dataset. Pre-trained models are leveraged to obtain a union of the COCO and MPII keypoints, and a heuristic is used to estimate additional spinal cord keypoints. This automated process creates an initial dataset that serves as a starting point for further refinement.

A subset of the initial dataset is then manually annotated using the CVAT annotation tool to refine the new spinal cord keypoints. This manually refined subset forms our base dataset.

To scale up the annotation process, we employ an active learning strategy. A neural network is trained on the base dataset and used to predict keypoints on the remaining images in the initial dataset. The model's uncertainty is evaluated, and the images with the highest uncertainty are selected for manual annotation. Once annotated, these images are added to the base dataset, incrementally improving its size and quality.

This active learning process is iteratively repeated, progressively reducing the pool of unannotated images, while enhancing the base dataset with the most informative examples. The process concludes when all images have been annotated, resulting in a robust and comprehensive extended 2D body keypoints dataset.

Getting Started

1. Clone the repository:

   git clone https://github.com/saifkhichi96/active-keypoint-extension.git
   cd active-keypoint-extension

2. Create a virtual environment and install the required packages:

   python3 -m venv venv
   source venv/bin/activate
   pip install -r requirements.txt

3. Install the latest version of MMPose by executing the following command:

   bash scripts/install_mmpose.sh

   This will install the latest version of MMPose from the official repository. MMPose is used to generate the initial dataset.

4. Download the COCO Keypoints 2017 dataset:

   bash scripts/download_coco.sh

   This will download the COCO Keypoints 2017 dataset from the official website and extract it to the data/coco directory.

Generating the Initial Dataset

The initial dataset is generated by executing the following command:

bash scripts/generate_initial_dataset.sh val2017
bash scripts/generate_initial_dataset.sh train2017

This will generate the initial dataset for the COCO validation and training sets, respectively. The generated dataset will be saved to the data/coco-extended directory. This will take a while to complete (approximately 1 hour), as it involves running the MMPose model on each image in the COCO dataset.

For finer control over the generation process, run the following command to see the available options:

python3 src/generate_initial_dataset.py --help

Specifically, the following options are available:

1. --min_bbox_overlap: The minimum IOU threshold for merging COCO and MPII detections. This value is used to match persons detected by COCO and MPII pose estimation models to each other. The default value is 0.7, which means we consider two bounding boxes to be the same person if they overlap by at least 70%.

2. --min_bbox_score: The minimum score for a bounding box to be considered 'good'. This value is used to filter out bounding boxes with low confidence scores. The default value is 0.5, which means we only consider bounding boxes with a confidence score of at least 50%.

3. --min_bbox_area: The minimum area of a bounding box in pixels. This value is used to filter out bounding boxes that are too small, which can reduce false positives. The default value is 32, which means we only consider bounding boxes with an area of at least 32x32 pixels.

4. --min_keypoint_score: The minimum score for a keypoint to be considered 'good'. This value is used to filter out keypoints with low confidence scores. The default value is 0.3, which means we only consider a keypoint to be 'good' if it has a confidence score of at least 30%.

5. --min_keypoints: The minimum number of 'good' keypoints required to keep a person. This value is used to filter out persons with too few keypoints, which can reduce false positives. The default value is 3, which means we only keep detections with at least 3 'good' keypoints.

Manually Annotating the Base Dataset

When the initial dataset has been generated, it is time to manually annotate a subset of the images to create the base dataset. This is done using the CVAT annotation tool.

Launching the CVAT Server

To launch the CVAT server, open Google Chrome and navigate to the following URL: http://mindgarage03.cs.uni-kl.de:8080/. This will open CVAT in your browser, and you will be able to log in using your CVAT credentials.

Please note that the CVAT server is only accessible from within the RPTU network. If you are not connected to the university network, you will need to use a VPN to access the CVAT server.

Setting up the CVAT Project

Once you have logged in to the CVAT server, you will need to create a new project and set up a skeleton with the extended keypoints. Please note that you only need to do this once. If a project has already been created, you can skip to the 'Importing the Initial Dataset' section.

To set up a new project, follow the instructions on your screen to create a new project and name it 'COCO Extended Keypoints'. Then, click 'Submit and Open' to create and open the project.

Setting up the Skeleton

To set up the skeleton, follow these steps:

1. Look for the "Setup skeleton" option on the Project page and click it.
2. In the "Label name" field, enter 'person'.
3. Choose the "Upload a skeleton from SVG" option from the icons on the left.
4. A file dialog will open; navigate to and select the data/cvat_skeleton_21.svg file to upload it.
5. CVAT will show a preview of the skeleton. Verify that the skeleton is correct, and then click the "Continue" button.

Creating the Tasks

Once you have created the project, create two new tasks with names 'val2017' and 'test2017' respectively. For each task, select all images in the 'data/coco/val2017' and 'data/coco/test2017' directories, respectively. Then, click 'Submit' to create the tasks.

Importing the Initial Dataset

To import the initial dataset, follow these steps:

1. Open the task you want to import the dataset into (either 'val2017' or 'test2017').
2. In the 'Actions' dropdown on the top-righ corner, select 'Upload annotations'. A dialog box will open.
3. In the dialog box, set the 'Import format' to 'COCO Keypoints 1.0'.
4. In the file field, select the data/coco-extended/val2017.zip or data/coco-extended/test2017.zip file, depending on the task you are importing the dataset into.
5. Click the 'Ok' button to import the dataset.

This will take several minutes to complete, and you will be notified when the import is complete. Please wait for the import to complete for each task before proceeding to the next step.

Annotating the Images

Once the initial dataset has been imported, you can start fine-tuning the annotations in the CVAT tool.

Re-training the Model

This section is still under construction.