-
Notifications
You must be signed in to change notification settings - Fork 314
Launch instances with ODCR (On Demand Capacity Reservations)
The following content has been moved to the official documentation. Please refer to Launch instances with ODCR (On-Demand Capacity Reservations) - ParallelCluster 3.x for updated instructions.
ODCRs can be configured as 'open' or 'targeted'. Open ODCRs will cover any instances that match the ODCR attributes (instance type, platform, Availability Zone) while targeted ODCRs need to be explicitly defined in the run-instances call. It’s possible to distinguish open/targeted ODCR, by executing describe-capacity-reservation.
Then there is the possibility to create ODCR associated with a PG, this reservation is called CPG ODCR (or PG/ODCR).
Multiple ODCRs can be grouped in a Resource Group, this can be used in the run-instances override to target multiple ODCR at the same time.
Open ODCR is supported out of the box by any ParallelCluster version. In an Open ODCR you don’t need to specify anything in ParallelCluster, the instances will be automatically selected. You can specify an existing Placement Group or enable it to permit ParallelCluster to create a new one. Note, however if you specify a Placement Group you might risk getting an Insufficient Capacity Exception if the reserved instances do not live on the same spine. If you need to use a Placement Group with your ODCR, see the CPG-ODCR link below.
Support for targeted ODCR has been added starting from ParallelCluster 3.1.1 and ParallelCluster 2.11.3. We have introduced a mechanism to override EC2 RunInstances parameters and therefore pass information about the reservation to use for each configured compute resource in ParallelCluster. This mechanism is compatible with the usage of targeted ODCR. For this case it is required to specify run-instances override configuration (described in the next sections).
If you’re using a targeted ODCR, you could specify a Placement Group and this should work, in any case you need to specify run-instances override configuration.
If AWS created a targeted ODCR for you, or you have a specific set of reserved instances, you cannot specify a Placement Group, because the rules configured by AWS could conflict with the Placement Group setting. So if a PG is required for your application, you need to ask for a CPG ODCR. In any case you need to specify run-instances override configuration.
If you’re using a CPG/ODCR, you have to specify the run-instances override configuration and you have to specify the same Placement Group in the cluster configuration.
Reserved instances are different by Capacity Reservation (ODCR) and there are 2 types of reserved instances. A regional Reserved Instance does not reserve capacity. A zonal Reserved Instance reserves capacity in the specified Availability Zone.
If you have regional Reserved instances there is no capacity reservation and you can get Insufficient Capacity Errors. If you have zonal Reserved Instances you have capacity reservation but there are no parameters you can specify in the run-instances api to use them.
Reserved instances are supported out of the box by any ParallelCluster version, you don't need to specify anything in ParallelCluster, the instances will be automatically selected.
When using Zonal Reserved Instances you should not specify a Placement Group in the configuration, you can get Insufficient Capacity Errors.
EC2 RunInstances parameters can be overridden for each compute resource configured in a cluster queue. In order to do so create the /opt/slurm/etc/pcluster/run_instances_overrides.json file on the head node of the cluster with the following content:
{
"${queue_name}": {
"${compute_resource_name}": {
${overrides}
},
...
},
...
}
where:
-
${queue_name}: is the name of the queue you want to apply overrides to -
${compute_resource_name}: is the compute resource you want to apply overrides to -
${overrides}: is an arbitrary Json object containing the list of RunInstances overrides to use for the specific combination of queue/instance-type. The overrides syntax needs to follow the same specifications documented in a run_instances boto3 call.
For example the following json configures the ODCR group group_arn to be used for p4d.24xlarge instances that are configured in queue my-queue and in my-compute-resource compute resource.
{
"my-queue": {
"my-compute-resource": {
"CapacityReservationSpecification": {
"CapacityReservationTarget": {
"CapacityReservationResourceGroupArn": "group_arn"
}
}
}
}
}
Once such Json file is generated, the ParallelCluster daemons responsible for cluster scaling will automatically pick up such configuration and start using it for instance launches. In order to confirm that the specified parameters are being used for instance provisioning you can look at the following log files: /var/log/parallelcluster/clustermgtd (in case of static capacity) or /var/log/parallelcluster/slurm_resume.log (in case of dynamic capacity). If the parameters are being properly used you will find a log entry containing: Found RunInstances parameters override. Launching instances with: <parameters_list>.
Create a resource group to group capacity
aws resource-groups create-group --name EC2CRGroup \
--configuration '{"Type":"AWS::EC2::CapacityReservationPool"}' '{"Type":"AWS::ResourceGroups::Generic", "Parameters": [{"Name": "allowed-resource-types", "Values": ["AWS::EC2::CapacityReservation"]}]}'
Add capacity reservations to the resource group, every time you create a new ODCR please add it to the Group Reservation. Please ensure to replace PLACEHOLDER_ACCOUNT_ID with your account ID, PLACEHOLDER_CAPACITY_RESERVATION with the ID of your capacity reservation and PLACEHOLDER_AWS_REGION with your AWS region ID (us-east-1 for example).
aws resource-groups group-resources --region PLACEHOLDER_AWS_REGION --group EC2CRGroup \
--resource-arns arn:aws:ec2:PLACEHOLDER_AWS_REGION:PLACEHOLDER_ACCOUNT_ID:capacity-reservation/PLACEHOLDER_CAPACITY_RESERVATION
Create a policy document on your local computer. Please ensure to replace PLACEHOLDER_ACCOUNT_ID with your account ID and PLACEHOLDER_AWS_REGION with your AWS region ID (us-east-1 for example).
cat > policy.json << EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "RunInstancesInCapacityReservation",
"Effect": "Allow",
"Action": "ec2:RunInstances",
"Resource": [
"arn:aws:ec2:PLACEHOLDER_AWS_REGION:PLACEHOLDER_ACCOUNT_ID:capacity-reservation/*",
"arn:aws:resource-groups:PLACEHOLDER_AWS_REGION:PLACEHOLDER_ACCOUNT_ID:group/*"
]
}
]
}
EOF
Create the IAM policy on your AWS account using the json file you created.
aws iam create-policy --policy-name RunInstancesInCapacityReservation —policy-document file://policy.json
Create the following post install script locally on the instance with the name postinstall.sh. Please ensure to replace PLACEHOLDER_ACCOUNT_ID with your account ID and PLACEHOLDER_AWS_REGION with your AWS region ID (us-east-1 for example).
#!/bin/bash
set -e
# Override run_instance attributes
cat > /opt/slurm/etc/pcluster/run_instances_overrides.json << EOF
{
"my-queue": {
"my-compute-resource": {
"CapacityReservationSpecification": {
"CapacityReservationTarget": {
"CapacityReservationResourceGroupArn": "arn:aws:resource-groups:PLACEHOLDER_AWS_REGION:PLACEHOLDER_ACCOUNT_ID:group/EC2CRGroup"
}
}
}
}
}
EOF
Upload the file to S3. Please replace PLACEHOLDER_NAME_BUCKET with a unique bucket name.
aws s3 mb s3://PLACEHOLDER_NAME_BUCKET
aws s3 cp postinstall.sh s3://PLACEHOLDER_NAME_BUCKET/postinstall.sh
Create the local cluster configuration replacing the placeholders.
Region: PLACEHOLDER_AWS_REGION
Image:
Os: alinux2
HeadNode:
InstanceType: c5.2xlarge
Ssh:
KeyName: PLACEHOLDER_NAME_OF_YOUR_SSH_KEY
Iam:
S3Access:
- BucketName: PLACEHOLDER_NAME_BUCKET
AdditionalIamPolicies:
- Policy: arn:aws:iam::PLACEHOLDER_AWS_ACCOUNT_ID:policy/RunInstancesInCapacityReservation
## This post-install script is executed after the node is configured.
## It is used to install scripts at boot time and specific configurations
## In the script below we are overriding the calls to RunInstance to force
## the provisioning of our my-queue partition to go through
## the On-Demand Capacity Reservation
CustomActions:
OnNodeConfigured:
Script: s3://PLACEHOLDER_NAME_BUCKET/postinstall.sh
Networking:
SubnetId: PLACEHOLDER_PUBLIC_SUBNET_IN_TARGET_AZ
Scheduling:
Scheduler: slurm
SlurmQueues:
- Name: my-queue
ComputeResources:
- MinCount: 0
MaxCount: 100
InstanceType: p4d.24xlarge
Name: my-compute-resource
Efa:
Enabled: true
Networking:
## PlacementGroup:
## Enabled: true ## Keep PG disabled if using targeted ODCR
SubnetIds:
- PLACEHOLDER_PRIVATE_SUBNET_IN_TARGET_AZ
Create the cluster as follows, assuming that cluster-config.yaml is the name of your config file and cluster-dl the name you picked for your cluster.
pcluster create-cluster --cluster-configuration cluster-config.yaml --cluster-name cluster-dl --region PLACEHOLDER_AWS_REGION
Once the cluster is created, the post-install script is executed in the head node. In the script we are creating the run_instances_overrides.json file and overriding the calls to RunInstance to force the provisioning of the partition to go through
the On-Demand Capacity Reservation.
The ParallelCluster daemons responsible for cluster scaling will automatically pick up such configuration and start using it for instance launches. In order to confirm that the specified parameters are being used for instance provisioning you can look at the following log files: /var/log/parallelcluster/clustermgtd (in case of static capacity - MinCount > 0) or /var/log/parallelcluster/slurm_resume.log (in case of dynamic capacity). If the parameters are being properly used you will find a log entry containing: Found RunInstances parameters override. Launching instances with: <parameters_list>.
The generated Json configuration can be updated at any time without stopping the compute fleet. After the changes are applied any new instance launch will use the updated configuration. In case you need to apply the updated configuration to running nodes you will have to recycle the nodes by forcing an instance termination and wait for ParallelCluster to replace those nodes. This can be achieved by either terminating the instance through EC2 or by setting the Slurm nodes in a DOWN or DRAIN state.
Example to set Slurm node to Down or DRAIN:
scontrol update nodename=my-queue-dy-my-compute-resource-1 state=down reason=your_reason
scontrol update nodename=my-queue-dy-my-compute-resource-1 state=drain reason=your_reason