PCluster 3.13.0 lfs hsm_restore degraded performance

[stefan-vaisala]

Required Info:

• AWS ParallelCluster version [e.g. 3.1.1]: 3.13.0 with base PCluster AL2023 AMI
• FSx lustre configuration from PCluster YAML:

  - Name: fs
    StorageType: FsxLustre
    MountDir: [redacted]
    FsxLustreSettings:
      StorageCapacity: 28800
      DeploymentType: PERSISTENT_1
      PerUnitStorageThroughput: 200
      ImportedFileChunkSize: 1024
      ImportPath: [redacted]
      WeeklyMaintenanceStartTime: "6:00:00"

Bug description and how to reproduce:
In our cluster, we spin up an Persistent 1 FSx for Lustre file system that is hooked up to an S3 bucket with ~345GB across ~37,000 objects. During our post install procedures, we warm the FSx lustre file system using find [...] -type f -print0 | xargs -0 --max-args=1 --max-procs=16 sudo lfs hsm_restore as executed from a head node instance that is a c7a.8xlarge.

This hsm_restore process typically takes 5-6 minutes to warm all 345GB and is consistent in timing from PCluster version 3.8.0 through 3.12.0. However, with PCluster 3.13.0, the hsm_restore process never completes (we've let it run 1 hour+) and times out the PCluster CloudFormation stack with "Create Failed" due to the HeadNode timeout. Looking at the FSx metrics, it looks like all the hsm_restore processes are not running on the file system, as the IOPS and throughput of the file system are effectively in "idle", with very little IOPS or network throughput. Further, it looks like the hsm_restore processes immediately go to "sleep" mode on the HeadNode. We also noticed that the lustre client version went from 2.15.3 in PCluster 3.12.0 to 2.15.6 in PCluster 3.13.0.

We are hoping you can provide some insight into this as we currently can't spin up clusters using PCluster 3.13.0 without removing the hsm_restore commands due to its extremely degraded performance. Thanks! - Stefan

[stefan-vaisala]

Quick update - I did a downgrade the lustre client to the same version as in PCluster 3.12. (version 2.15.13) and created a custom PCluster 3.13.0 AMI with that downgraded version. Running hsm_restore using that downgraded client did not change the degraded hsm_restore performance.

[stefan-vaisala]

Another update as we are still debugging on our end. We tested the following to determine where the issue might be occurring:

1. Disable lfs hsm_restore in our post-install shell script (comment out the calls).
2. Create a cluster, still run the remaining post-install commands/configurations, and let the cluster get to CREATE_COMPLETE state in CloudFormation.
3. Login to the cluster, switch to root user, and create a shell script with the identical lfs hsm_restore calls as in the post-install shell script.
4. Run the shell script created in (3) with just the lfs hsm_restore commands as root user.

Following these steps, the lfs hsm_restore commands run as expected, finishing in 5-6 minutes. So, something is happening with the lfs hsm_restore commands and how the post-install shell script is being called/executed. @hanwen-pcluste / @demartinofra - did anything change with PCluster 3.13.0 with how the post-install shell scripts are called? It seems like the lfs hsm_restore processes are being blocked/throttled when called from within a post-install script as part of PCluster's 3.13.0 spin up process.