Un workflow manuel sur self-hosted host pour passer d'un nouveau modèle entraîné à une optimisation des seuils de lidar-prod

[CharlesGaydon]

L'idée est, à partir d'une image de myria3d incluant un modèle et sa configuration, de pouvoir facilement actualiser les seuils dans lidar-prod, sur la machine locale qui a accès aux données de val et de test.

J'ai drafté quelque chose avec ChatGPT, il y a des tas d'erreurs au niveau des chemins de fichier, mais ça permet d'illustrer ce que je veux dire :

name: Data Science Workflow
on:
  workflow_dispatch:
    inputs:
      docker_image_myria3D:
        description: 'Full Docker image reference for myria3D'
        required: true
        default: 'my-registry/myria3D:latest'
      docker_image_lidar_prod:
        description: 'Full Docker image reference for lidar_prod'
        required: true
        default: 'my-registry/lidar_prod:latest'
      input_dir:
        description: 'Local input directory for the tasks'
        required: true
        default: '/path/to/local/input/dir'
      output_dir:
        description: 'Local output directory for the tasks'
        required: true
        default: '/path/to/local/output/dir'

jobs:
  run_validation:
    runs-on: self-hosted
    steps:
      - name: Run myria3D on validation dataset
        run: |
          docker run \
            -v ${{ inputs.input_dir }}:/inputs/ \
            -v ${{ inputs.output_dir }}:/outputs/ \
            --ipc=host \
            --gpus=all \
            --shm-size=2gb \
            ${{ inputs.docker_image_myria3D }} \
            task.task_name=predict \
            predict.src_las=/inputs/validation_cloud.las \
            predict.output_dir=/outputs/ \
            predict.gpus=0 \
            datamodule.batch_size=4

      - name: Run lidar_prod on validation dataset
        run: |
          docker run \
            -v ${{ inputs.input_dir }}:/inputs/ \
            -v ${{ inputs.output_dir }}:/outputs/ \
            ${{ inputs.docker_image_lidar_prod }} \
            python lidar_prod/run.py \
            paths.src_las=/inputs/validation_cloud.laz \
            paths.output_dir=/outputs/ \
            task=optimize_building \
            building_validation.optimization.todo='prepare+optimize+evaluate+update' \
            building_validation.optimization.paths.input_las_dir=/inputs/labelled_val_dataset/ \
            building_validation.optimization.paths.results_output_dir=/outputs/

  run_test:
    runs-on: self-hosted
    steps:
      - name: Run myria3D on test dataset
        run: |
          docker run \
            -v ${{ inputs.input_dir }}:/inputs/ \
            -v ${{ inputs.output_dir }}:/outputs/ \
            --ipc=host \
            --gpus=all \
            --shm-size=2gb \
            ${{ inputs.docker_image_myria3D }} \
            task.task_name=predict \
            predict.src_las=/inputs/test_cloud.las \
            predict.output_dir=/outputs/ \
            predict.gpus=0 \
            datamodule.batch_size=4

      - name: Run lidar_prod on test dataset
        run: |
          docker run \
            -v ${{ inputs.input_dir }}:/inputs/ \
            -v ${{ inputs.output_dir }}:/outputs/ \
            ${{ inputs.docker_image_lidar_prod }} \
            python lidar_prod/run.py \
            paths.src_las=/inputs/test_cloud.laz \
            paths.output_dir=/outputs/ \
            task=optimize_building \
            building_validation.optimization.todo='prepare+evaluate+update' \
            building_validation.optimization.paths.input_las_dir=/inputs/labelled_test_dataset/ \
            building_validation.optimization.paths.results_output_dir=/outputs/ \
            building_validation.optimization.paths.building_validation_thresholds_pickle=/inputs/optimized_thresholds.pickle

[leavauchier]

Quelque chose de similaire est mis en place ici : #110