AWS ParallelCluster v3.0.1

We're excited to announce the release of AWS ParallelCluster 3.0.1

Upgrade

How to upgrade?

sudo pip install --upgrade aws-parallelcluster

3.0.1

ENHANCEMENTS

• Add pcluster3-config-converter CLI command to convert cluster configuration from ParallelCluster 2 to ParallelCluster 3 version.
• The region parameter is now retrieved from the provider chain, thus supporting the use of profiles and defaults specified in the ~/.aws/config file.
• Export ParallelClusterApiInvokeUrl and ParallelClusterApiUserRole in CloudFormation API Stack so they can be used by cross-stack references.

CHANGES

• Drop support for SysVinit. Only Systemd is supported.
• Include tags from cluster configuration file in the RunInstances dry runs performed during configuration validation.
• Allow '*' character in the configuration of S3Access/BucketName.

BUG FIXES

• Pin to the transitive dependencies resulting from the dependency on connexion.
• Fix cleanup of ECR resources when API infrastructure template is deleted.
• Fix supervisord service not enabled on Ubuntu. This was causing supervisord not to be started on instance reboot.
• Update ca-certificates package during AMI build time and have Cinc use the updated CA certificates bundle.
• Close stderr before exiting from pcluster CLI commands to avoid BrokenPipeError for processes that close the other end of the stdout pipe.

AWS ParallelCluster v3.0.0

We're excited to announce the release of AWS ParallelCluster 3.0.0

Upgrade

How to upgrade?

sudo pip install --upgrade aws-parallelcluster

3.0.0

ENHANCEMENTS

• Add support for pcluster actions (e.g., create-cluster, update-cluster, delete-cluster) through HTTP endpoints
  with Amazon API Gateway.
• Revamp custom AMI creation and management by leveraging EC2 Image Builder. This also includes the implementation of
  build-image, delete-image, describe-image and list-image commands to manage custom ParallelCluster images.
• Add list-official-images command to describe ParallelCluster official AMIs.
• Add export-cluster-logs, list-cluster-logs and get-cluster-log-events commands to retrieve both CloudWatch Logs
  and CloudFormation Stack Events. Add export-image-logs, list-image-logs and get-image-log-events commands to
  retrieve both Image Builder Logs and CloudFormation Stack Events.
• Enable the possibility to restart / reboot the head node also for instance types with
  instance store.
  Those operations remain anyway managed by the user that is responsible for the status of the cluster while operating
  on the head node, e.g. stopping the compute fleet first.
• Add support to use an existing Private Route53 Hosted Zone when using Slurm as scheduler.
• Add the possibility to configure the instance profile as alternative to configuring the IAM role for the head and for
  each compute queue.
• Add the possibility to configure IAM role, profile and policies for head node and for each compute queue.
• Add possibility to configure different security groups for each queue.
• Allow full control on the name of CloudFormation stacks created by ParallelCluster by removing the parallelcluster-
  prefix.
• Add multiple queues and compute resources support for pcluster configure when the scheduler is Slurm.
• Add prompt for availability zone in pcluster configure automated subnets creation.
• Add configuration HeadNode / Imds / Secured to enable/disable restricted access to Instance Metadata Service (IMDS).
• Implement scaling protection mechanism with Slurm scheduler: compute fleet is automatically set to 'PROTECTED'
  state in case recurrent failures are encountered when provisioning nodes.
• Add --suppress-validators and --validation-failure-level parameters to create and update commands.
• Add support for associating an existing Elastic IP to the head node.
• Extend limits for supported number of Slurm queues (10) and compute resources (5).
• Encrypt root EBS volumes and shared EBS volumes by default. Note that if the scheduler is AWS Batch, the root volumes
  of the compute nodes cannot be encrypted by ParallelCluster.

CHANGES

• Upgrade EFA installer to version 1.13.0
  • EFA configuration: efa-config-1.9
  • EFA profile: efa-profile-1.5
  • EFA kernel module: efa-1.13.0
  • RDMA core: rdma-core-35
  • Libfabric: libfabric-1.13.0
  • Open MPI: openmpi40-aws-4.1.1-2
• Upgrade NICE DCV to version 2021.1-10851.
• Upgrade Slurm to version 20.11.8.
• Upgrade NVIDIA driver to version 470.57.02.
• Upgrade CUDA library to version 11.4.0.
• Upgrade Cinc Client to version 17.2.29.
• Upgrade Python runtime used by Lambda functions in AWS Batch integration to python3.8.
• Remove support for SGE and Torque schedulers.
• Remove support for CentOS8.
• Change format and syntax of the configuration file to be used to create the cluster, from ini to YAML. A cluster configuration
  file now only includes the definition of a single cluster.
• Remove --cluster-template, --extra-parameters and --tags parameters for the create command.
• Remove --cluster-template, --extra-parameters, --reset-desired and --yes parameters for the update command.
• Remove --config parameter for delete, status, start, stop, instances and list commands.
• Remove possibility to specify aliases for ssh command in the configuration file.
• Distribute AWS Batch commands: awsbhosts, awsbkill, awsbout, awsbqueues, awsbstat and awsbsub as a
  separate aws-parallelcluster-awsbatch-cli PyPI package.
• Add timestamp suffix to CloudWatch Log Group name created for the cluster.
• Remove pcluster-config CLI utility.
• Remove amis.txt file.
• Remove additional EBS volume attached to the head node by default.
• Change NICE DCV session storage path to /home/{UserName}.
• Create a single ParallelCluster S3 bucket for each AWS region rather than for each cluster.
• Adopt inclusive language
  • Rename MasterServer to HeadNode in CLI outputs.
  • Rename variable exported in the AWS Batch job environment from MASTER_IP to PCLUSTER_HEAD_NODE_IP.
  • Rename all CFN outputs from Master* to HeadNode*.
  • Rename NodeType and tags from Master to HeadNode.
• Rename tags (Note: the following tags are crucial for ParallelCluster scaling logic):
  • aws-parallelcluster-node-type -> parallelcluster:node-type
  • ClusterName -> parallelcluster:cluster-name
  • aws-parallelcluster-attributes -> parallelcluster:attributes
  • Version -> parallelcluster:version
• Remove tag: Application.
• Remove runtime creation method
  of custom ParallelCluster AMIs.
• Retain CloudWatch logs on cluster deletion by default. If you want to delete the logs during cluster deletion, set
  Monitoring / Logs / CloudWatch / RetainOnDeletion to False in the configuration file.
• Remove instance store software encryption option (encrypted_ephemeral) and rely on default hardware encryption provided
  by NVMe instance store volumes.
• Add tag 'Name' to every shared storage with the value specified in the shared storage name config.
• Remove installation of MPICH and FFTW packages.
• Remove Ganglia support.

AWS ParallelCluster v2.11.2

We're excited to announce the release of AWS ParallelCluster 2.11.2

Upgrade

How to upgrade?

sudo pip install --upgrade aws-parallelcluster

2.11.2

CHANGES

• When using a custom AMI with a preinstalled EFA package, no actions are taken at node bootstrap time in case GPUDirect RDMA is enabled. The original EFA package deployment is preserved as during the createami process.
• Upgrade EFA installer to version 1.13.0
  • Update rdma-core to v35.0.
  • Update libfabric to v1.13.0amzn1.0.

BUG FIXES

• Lock the version of nvidia-fabricmanager package to the installed NVIDIA drivers to prevent updates and misalignments.
• Slurm: fix issue that prevented powering-up nodes to be correctly reset after a stop and start of the cluster.

AWS ParallelCluster v2.11.1

We're excited to announce the release of AWS ParallelCluster 2.11.1

Upgrade

How to upgrade?

sudo pip install --upgrade aws-parallelcluster

CHANGES

• Restore noatime option, which has positive impact on the performances of NFS filesystem.
• Upgrade EFA installer to version 1.12.3
  • EFA configuration: efa-config-1.9 (from efa-config-1.8-1)
  • EFA kernel module: efa-1.13.0 (from efa-1.12.3)

BUG FIXES

• Pin to version 1.247347 of the CloudWatch agent due to performance impact of latest CW agent version 1.247348.
• Avoid failures when building SGE using instance type with vCPU >=32.

AWS ParallelCluster v2.11.0

We're excited to announce the release of AWS ParallelCluster 2.11.0

Upgrade

How to upgrade?

sudo pip install --upgrade aws-parallelcluster

ENHANCEMENTS

• Add support for Ubuntu 20.04.
• Add support for using FSx Lustre in subnet with no internet access.
• Add support for building custom Centos 7 AMIs on ARM.
• Add support for FSx Lustre DataCompressionType feature.
• Add validation to prevent using a cluster_resource_bucket that is in a different region than the cluster.
• Install SSM agent on CentOS 7 and 8.
• Add support for security_group_id in packer custom builders. Customers can export AWS_SECURITY_GROUP_ID environment variable to specify security group for custom builders when building custom AMIs.
• SGE: always use shortname as hostname filter with qstat. This will make nodewatcher more robust when using custom DHCP option, where the full hostname seen by SGE might differ from the hostname returned from EC2 metadata(local-hostname).
• Transition from IMDSv1 to IMDSv2.

CHANGES

• Removed support for Ubuntu 16.04 (ubuntu1604).
• Removed support for Amazon Linux (alinux). Amazon Linux 2 (alinux2) remains fully supported.
  Amazon Linux is no longer supported.
• Make key_name parameter optional to support cluster configurations without a key pair.
• Remove support for Python versions < 3.6.
  • Remove dependency on future package and __future__ module.
• Root volume size increased from 25GB to 35GB on all AMIs. Minimum root volume size is now 35GB.
• Add sanity check to prevent cluster creation in an AWS region not officially supported by ParallelCluster.
• Restrict IAM permissions to only allow cluster IAM instance role to launch instances via run-instances in cluster compute subnet.
• Upgrade EFA installer to version 1.12.2
  • EFA configuration: efa-config-1.8-1 (from efa-config-1.7)
  • EFA profile: efa-profile-1.5-1 (from efa-profile-1.4)
  • EFA kernel module: efa-1.12.3 (from efa-1.10.2)
  • RDMA core: rdma-core-32.1amzn (from rdma-core-31.2amzn)
  • Libfabric: libfabric-1.11.2amzon1.1-1 (from libfabric-1.11.1amzn1.0)
  • Open MPI: openmpi40-aws-4.1.1-2 (from openmpi40-aws-4.1.0)
• Upgrade Slurm to version 20.11.7.
  • Update slurmctld and slurmd systemd unit files according to latest provided by slurm.
  • Add new SlurmctldParameters, power_save_min_interval=30, so power actions will be processed every 30 seconds.
  • Add new SlurmctldParameters, cloud_reg_addrs, which will reset a node's NodeAddr automatically on power_down.
  • Specify instance GPU model as GRES GPU Type in gres.conf, instead of previous hardcoded value Type=tesla for all GPU.
• Upgrade Arm Performance Libraries (APL) to version 21.0.0.
• Upgrade NICE DCV to version 2021.1-10557.
• Upgrade NVIDIA driver to version 460.73.01.
• Upgrade CUDA library to version 11.3.0.
• Upgrade NVIDIA Fabric manager to nvidia-fabricmanager-460.
• Install ParallelCluster AWSBatch CLI in dedicated python3 virtual env.
• Upgrade Python version used in ParallelCluster virtualenvs from version 3.6.13 to version 3.7.10.
• Upgrade Cinc Client to version 16.13.16.
• Upgrade third-party cookbook dependencies:
  • apt-7.4.0 (from apt-7.3.0)
  • iptables-8.0.0 (from iptables-7.1.0)
  • line-4.0.1 (from line-2.9.0)
  • openssh-2.9.1 (from openssh-2.8.1)
  • pyenv-3.4.2 (from pyenv-3.1.1)
  • selinux-3.1.1 (from selinux-2.1.1)
  • ulimit-1.1.1 (from ulimit-1.0.0)
  • yum-6.1.1 (from yum-5.1.0)
  • yum-epel-4.1.2 (from yum-epel-3.3.0)
• Drop lightdm package install from Ubuntu 18.04 DCV installation process.

BUG FIXES

• Use ICP-compliant AL2 repo URLs when building Docker images in China
• Fix a bug that caused clustermgtd to not immediately replace instances with failed status check that are in replacement process.

AWS ParallelCluster v2.10.4

We're excited to announce the release of AWS ParallelCluster 2.10.4

Upgrade

How to upgrade?

sudo pip install --upgrade aws-parallelcluster

CHANGES

• Upgrade Slurm to version 20.02.7.

AWS ParallelCluster v2.10.3

We're excited to announce the release of AWS ParallelCluster 2.10.3

Upgrade

How to upgrade?

sudo pip install --upgrade aws-parallelcluster

ENHANCEMENTS

• Enable support for ARM instances in China and GovCloud regions when using Ubuntu 18.04 or Amazon Linux 2.
• Add validation for cluster_type configuration parameter in cluster section
• Add validation for compute_type configuration parameter in queue section

CHANGES

• Upgrade EFA installer to version 1.11.2
  • EFA configuration: efa-config-1.7 (no change)
  • EFA profile: efa-profile-1.4 (from efa-profile-1.3)
  • EFA kernel module: efa-1.10.2 (no change)
  • RDMA core: rdma-core-31.2amzn (no change)
  • Libfabric: libfabric-1.11.1amzn1.0 (no change)
  • Open MPI: openmpi40-aws-4.1.0 (no change)

BUG FIXES

• Fix issue with awsbsub command when setting environment variables for the job submission

AWS ParallelCluster v2.10.2

We're excited to announce the release of AWS ParallelCluster 2.10.2

Upgrade

How to upgrade?

sudo pip install --upgrade aws-parallelcluster

ENHANCEMENTS

• Improve cluster config validation by using cluster target AMI when invoking RunInstances in dryrun mode.
• Improve configuration procedure for the Munge service.

CHANGES

• Update Python version used in ParallelCluster virtualenvs from version 3.6.9 to version 3.6.13.

BUG FIXES

• Fix sanity checks with ARM instance types by using cluster AMI when performing validation.
• Fix enable_efa parameter validation when using Centos8 and Slurm or ARM instances.
• Use non interactive apt update command when building custom Ubuntu AMIs.
• Fix encrypted_ephemeral = true when using Alinux2 or CentOS8.

AWS ParallelCluster v2.10.1

We're excited to announce the release of AWS ParallelCluster 2.10.1

Upgrade

How to upgrade?

sudo pip install --upgrade aws-parallelcluster

ENHANCEMENTS

• Add support for me-south-1 region (Bahrein), af-south-1 region (Cape Town) and eu-south-1 region (Milan)
  • At the time of this version launch:
    • Amazon FSx for Lustre and ARM instance types are not supported in me-south-1, af-south-1 and eu-south-1
    • AWS Batch is not supported in af-south-1
    • EBS io2 is not supported in af-south-1 and eu-south-1
• Install Arm Performance Libraries (APL) 20.2.1 on ARM AMIs (CentOS8, Alinux2, Ubuntu1804).
• Install EFA kernel module on ARM instances with alinux2 and ubuntu1804. This enables support for c6gn instances.
• Add support for io2 and gp3 EBS volume type.
• Add iam_lambda_role parameter under cluster section to enable the possibility to specify an existing IAM role to
  be used by AWS Lambda functions in CloudFormation. When using sge, torque, or slurm as the scheduler, pcluster will not create any IAM role if both ec2_iam_role and iam_lambda_role are provided.
• Improve robustness of a Slurm cluster when clustermgtd is down.
• Configure NFS threads to be max(8, num_cores) for performance. This enhancement will not take effect on Ubuntu 16.04.
• Optimize calls to DescribeInstanceTypes EC2 API when validating cluster configuration.

CHANGES

• Upgrade EFA installer to version 1.11.1.
  • EFA configuration: efa-config-1.7 (from efa-config-1.5)
  • EFA profile: efa-profile-1.3 (from efa-profile-1.1)
  • EFA kernel module: efa-1.10.2 (no change)
  • RDMA core: rdma-core-31.2amzn (from rdma-core-31.amzn0)
  • Libfabric: libfabric-1.11.1amzn1.0 (from libfabric-1.11.1amzn1.1)
  • Open MPI: openmpi40-aws-4.1.0 (from openmpi40-aws-4.0.5)
• Upgrade Intel MPI to version U8.
• Upgrade NICE DCV to version 2020.2-9662.
• Set default systemd runlevel to multi-user.target on all OSes during ParallelCluster official AMI creation.
  The runlevel is set to graphical.target on head node only when DCV is enabled. This prevents the execution of graphical services, such as x/gdm, when they are not required.
• Download Intel MPI and HPC packages from S3 rather than Intel yum repos.
• Change the default of instance types from the hardcoded t2.micro to the free tier instance type (t2.micro or t3.micro dependent on region). In regions without free tier, the default is t3.micro.
• Enable support for p4d as head node instance type (p4d was already supported as compute node in 2.10.0).
• Pull Amazon Linux Docker images from public ECR when building docker image for awsbatch scheduler.
• Increase max retry attempts when registering Slurm nodes in Route53.

BUG FIXES

• Fix pcluster createami for Ubuntu 1804 by downloading SGE sources from Debian repository and not from the EOL Ubuntu 19.10.
• Remove CloudFormation DescribeStacks API call from AWS Batch Docker entrypoint. This removes the risk of job failures due to CloudFormation throttling.
• Mandate the presence of vpc_settings, vpc_id, master_subnet_id in the config file to avoid unhandled exceptions.
• Set the default EBS volume size to 500 GiB when volume type is st1 or sc1.
• Fix installation of Intel PSXE package on CentOS 7 by using yum4.
• Fix routing issues with multiple Network Interfaces on Ubuntu 18.04.

AWS ParallelCluster v2.10.0

We're excited to announce the release of AWS ParallelCluster 2.10.0.

Upgrade

How to upgrade?

sudo pip install --upgrade aws-parallelcluster

ENHANCEMENTS

• Add support for CentOS 8 in all Commercial regions.
• Add support for P4d instance type as compute node.
• Add the possibilty to enable NVIDIA GPUDirect RDMA support on EFA by using the new enable_efa_gdr configuration
  parameter.
• Enable support for NICE DCV in GovCloud regions.
• Enable support for AWS Batch scheduler in GovCloud regions.
• FSx Lustre:
  • Add possibility to configure Auto Import policy through the new auto_import_policy parameter.
  • Add support to HDD storage type and the new storage_type and drive_cache_type configuration parameters.
• Create a CloudWatch Dashboard for the cluster, named <clustername>-<region>, including head node EC2 metrics and
  cluster logs. It can be disabled by configuring the enable parameter in the dashboard section.
• Add -r/-region arg to pcluster configure command. If this arg is provided, configuration will
  skip region selection.
• Add -r/-region arg tossh and dcv connect commands.
• Add cluster_resource_bucket parameter under cluster section to allow the user to specify an existing S3 bucket.
• createami:
  • Add validation step to fail when using a base AMI created by a different version of ParallelCluster.
  • Add validation step for AMI creation process to fail if the selected OS and the base AMI OS are not consistent.
  • Add --post-install parameter to use a post installation script when building an AMI.
  • Add the possibility to use a ParallelCluster base AMI.
• Add possibility to change tags when performing a pcluster update.
• Add new all_or_nothing_batch configuration parameter for slurm_resume script. When True, slurm_resume will
  succeed only if all the instances required by all the pending jobs in Slurm will be available.
• Enable queue resizing on update without requiring to stop the compute fleet. Stopping the compute fleet is only
  necessary when existing instances risk to be terminated.
• Add validator for EBS volume size, type and IOPS.
• Add validators for shared_dir parameter when used in both cluster and ebs sections.
• Add validator cfn_scheduler_slots key in the extra_json parameter.

CHANGES

• CentOS 6 is no longer supported.
• Upgrade EFA installer to version 1.10.1
  • EFA configuration: efa-config-1.5 (from efa-config-1.4)
  • EFA profile: efa-profile-1.1 (from efa-profile-1.0.0)
  • EFA kernel module: efa-1.10.2 (from efa-1.6.0)
  • RDMA core: rdma-core-31.amzn0 (from rdma-core-28.amzn0)
  • Libfabric: libfabric-1.11.1amzn1.1 (from libfabric-1.10.1amzn1.1)
  • Open MPI: openmpi40-aws-4.0.5 (from openmpi40-aws-4.0.3)
  • Unifies installer runtime options across x86 and aarch64
  • Introduces -g/--enable-gdr switch to install packages with GPUDirect RDMA support
  • Updates to OMPI collectives decision file packaging, migrated from efa-config to efa-profile
  • Introduces CentOS 8 support
• Upgrade NVIDIA driver to version 450.80.02.
• Install NVIDIA Fabric manager to enable NVIDIA NVSwitch on supported platforms.
• Remove default region us-east-1. After the change, pcluster will adhere to the following lookup order for region:
  1. -r/--region arg.
  2. AWS_DEFAULT_REGION environment variable.
  3. aws_region_name in ParallelCluster configuration file.
  4. region in AWScli configuration file.
• Slurm: change SlurmctldPort to 6820-6829 to not overlap with default slurmdbd port (6819).
• Slurm: add compute_resource name and efa as node features.
• Remove validation on ec2_iam_role parameter.
• Improve retrieval of instance type info by using DescribeInstanceType API.
• Remove custom_awsbatch_template_url configuration parameter.
• Upgrade pip to latest version in virtual environments.
• Upgrade image used by CodeBuild environment when building container images for Batch clusters, from
  aws/codebuild/amazonlinux2-x86_64-standard:1.0 to aws/codebuild/amazonlinux2-x86_64-standard:3.0.

BUG FIXES

• Retrieve the right number of compute instance slots when instance type is updated.
• Include user tags in compute nodes and EBS volumes.
• Fix pcluster status output when head node is stopped.
• pcluster update:
  • Fix issue when tags are specified but not changed.
  • Fix issue when the cluster section label changed.
  • Fix issue when shared_dir and ebs_settings are both configured in the cluster section.
  • Fix cluster and cfncluster compatibility in extra_json parameter.
• Fix pcluster configure to avoid using default/initial values for internal parameter initialization.
• Fix pre/post install script arguments management when using double quotes.
• Fix a bug that was causing clustermgtd and computemgtd sleep interval to be incorrectly computed when
  system timezone is not set to UTC.
• Fix queue name validator to properly check for capital letters.
• Fix enable_efa parameter validation for queue section.
• Fix CloudWatch Log Group creation for AWS Lambda functions handling CloudFormation Custom Resources.