Skip to content

Commit 91442e2

Browse files
hanwen-clusterhanwen-cluster
committed
Remove berkshelf
Berkshelf is a dependency manager for Chef cookbooks. With it, you can depend on community cookbooks and include them in your workflow. Berkshelf is not useful when all cookbooks are local. ParallelCluster cookbook has all third party cookbooks ready locally. Reducing software we depend on improves stability of the product. (e.g we have one known issue related to berkshelf https://github.com/aws/aws-parallelcluster/wiki/(3.8.0-%E2%80%90-3.9.3)-ParallelCluster-Build-Image-Failing-during-Installation-of-Minitar-Ruby-Gem-Dependency)
1 parent 3eb99d2 commit 91442e2

File tree

9 files changed

+9
-58
lines changed

9 files changed

+9
-58
lines changed

‎CHANGELOG.md

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,8 @@ CHANGELOG
66

77
**CHANGES**
88
- Ubuntu 20.04 is no longer supported.
9-
- Support DCV on AL2023
9+
- Support DCV on Amazon Linux 2023.
10+
- Remove `berkshelf`. All cookbooks are local and do not need `berkshelf` dependency management.
1011

1112
**BUG FIXES**
1213
- Fix an issue where Security Group validation failed when a rule contained both IPv4 ranges (IpRanges) and security group references (UserIdGroupPairs).

‎cli/src/pcluster/constants.py

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -165,7 +165,6 @@
165165
"parallelcluster": "3.14.0",
166166
"cookbook": "aws-parallelcluster-cookbook-3.14.0",
167167
"chef": "18.4.12",
168-
"berkshelf": "8.0.7",
169168
"ami": "dev",
170169
}
171170

‎cli/src/pcluster/resources/compute_node/user_data.sh

Lines changed: 1 addition & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -80,14 +80,7 @@ write_files:
8080
{
8181
mkdir /tmp/cookbooks
8282
cd /tmp/cookbooks
83-
tar -xzf /etc/chef/aws-parallelcluster-cookbook.tgz
84-
HOME_BAK="${!HOME}"
85-
export HOME="/tmp"
86-
for d in `ls /tmp/cookbooks`; do
87-
cd /tmp/cookbooks/$d
88-
LANG=en_US.UTF-8 /opt/cinc/embedded/bin/berks vendor /etc/chef/cookbooks --delete || error_exit 'Vendoring cookbook failed.'
89-
done;
90-
export HOME="${!HOME_BAK}"
83+
tar -xzf /etc/chef/aws-parallelcluster-cookbook.tgz --strip-components 1
9184
}
9285

9386
function publish_startup_time
@@ -131,7 +124,6 @@ write_files:
131124
export parallelcluster_version=aws-parallelcluster-${ParallelClusterVersion}
132125
export cookbook_version=${CookbookVersion}
133126
export chef_version=${ChefVersion}
134-
export berkshelf_version=${BerkshelfVersion}
135127
if [ -f /opt/parallelcluster/.bootstrapped ]; then
136128
installed_version=$(cat /opt/parallelcluster/.bootstrapped)
137129
if [ "${!cookbook_version}" != "${!installed_version}" ]; then

‎cli/src/pcluster/resources/head_node/user_data.sh

Lines changed: 1 addition & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -77,14 +77,7 @@ function vendor_cookbook
7777
{
7878
mkdir /tmp/cookbooks
7979
cd /tmp/cookbooks
80-
tar -xzf /etc/chef/aws-parallelcluster-cookbook.tgz
81-
HOME_BAK="${!HOME}"
82-
export HOME="/tmp"
83-
for d in `ls /tmp/cookbooks`; do
84-
cd /tmp/cookbooks/$d
85-
LANG=en_US.UTF-8 /opt/cinc/embedded/bin/berks vendor /etc/chef/cookbooks --delete || error_exit 'Vendoring cookbook failed.'
86-
done;
87-
export HOME="${!HOME_BAK}"
80+
tar -xzf /etc/chef/aws-parallelcluster-cookbook.tgz --strip-components 1
8881
}
8982
[ -f /etc/profile.d/proxy.sh ] && . /etc/profile.d/proxy.sh
9083

@@ -117,7 +110,6 @@ fi
117110
export parallelcluster_version=aws-parallelcluster-${ParallelClusterVersion}
118111
export cookbook_version=${CookbookVersion}
119112
export chef_version=${ChefVersion}
120-
export berkshelf_version=${BerkshelfVersion}
121113
if [ -f /opt/parallelcluster/.bootstrapped ]; then
122114
installed_version=$(cat /opt/parallelcluster/.bootstrapped)
123115
if [ "${!cookbook_version}" != "${!installed_version}" ]; then

‎cli/src/pcluster/resources/imagebuilder/parallelcluster.yaml

Lines changed: 1 addition & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -6,9 +6,6 @@ constants:
66
- ChefVersion:
77
type: string
88
value: 18.4.12
9-
- BerkshelfVersion:
10-
type: string
11-
value: 8.0.7
129
- FailExitCode:
1310
type: string
1411
value: 1
@@ -313,13 +310,6 @@ phases:
313310
ln -sf ${!CA_CERTS_FILE} /opt/cinc/embedded/ssl/certs/cacert.pem
314311
fi
315312
316-
curl --retry 3 -L -o gems.tgz https://${AWS::Region}-aws-parallelcluster.s3.${AWS::Region}.${AWS::URLSuffix}/archives/dependencies/ruby/gems.tgz
317-
tar -xf gems.tgz
318-
319-
cd vendor/cache
320-
/opt/cinc/embedded/bin/gem install --no-document minitar:0.9
321-
/opt/cinc/embedded/bin/gem install --local --no-document berkshelf:{{ BerkshelfVersion }}
322-
323313
# Download and vendor Cookbook
324314
- name: DownloadCookbook
325315
action: ExecuteBash
@@ -331,15 +321,7 @@ phases:
331321
332322
curl --retry 3 -L -o /etc/chef/aws-parallelcluster-cookbook.tgz "{{ build.CookbookUrl.outputs.stdout }}"
333323
334-
mkdir -p /tmp/cookbooks
335-
cd /tmp/cookbooks
336-
tar -xzf /etc/chef/aws-parallelcluster-cookbook.tgz
337-
338-
export HOME="/tmp"
339-
for dir in $(ls /tmp/cookbooks); do
340-
cd /tmp/cookbooks/${!dir}
341-
LANG=en_US.UTF-8 sudo /opt/cinc/embedded/bin/berks vendor /etc/chef/cookbooks --delete || (echo 'Vendoring cookbook failed.' && exit {{ FailExitCode }})
342-
done;
324+
cd /etc/chef && tar -xzf /etc/chef/aws-parallelcluster-cookbook.tgz --strip-components 1 && rm -f aws-parallelcluster-cookbook.tgz
343325
344326
- name: CreatingChefClientFile
345327
action: CreateFile

‎cli/src/pcluster/resources/login_node/user_data.sh

Lines changed: 2 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -67,14 +67,7 @@ write_files:
6767
{
6868
mkdir /tmp/cookbooks
6969
cd /tmp/cookbooks
70-
tar -xzf /etc/chef/aws-parallelcluster-cookbook.tgz
71-
HOME_BAK="${!HOME}"
72-
export HOME="/tmp"
73-
for d in `ls /tmp/cookbooks`; do
74-
cd /tmp/cookbooks/$d
75-
LANG=en_US.UTF-8 /opt/cinc/embedded/bin/berks vendor /etc/chef/cookbooks --delete || error_exit 'Vendoring cookbook failed.'
76-
done;
77-
export HOME="${!HOME_BAK}"
70+
tar -xzf /etc/chef/aws-parallelcluster-cookbook.tgz --strip-components 1
7871
}
7972

8073
export PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/opt/aws/bin
@@ -113,7 +106,7 @@ write_files:
113106
export parallelcluster_version=aws-parallelcluster-${ParallelClusterVersion}
114107
export cookbook_version=${CookbookVersion}
115108
export chef_version=${ChefVersion}
116-
export berkshelf_version=${BerkshelfVersion}
109+
117110
if [ -f /opt/parallelcluster/.bootstrapped ]; then
118111
installed_version=$(cat /opt/parallelcluster/.bootstrapped)
119112
if [ "${!cookbook_version}" != "${!installed_version}" ]; then

‎cli/src/pcluster/templates/cdk_builder_utils.py

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -94,7 +94,6 @@ def get_common_user_data_env(node: Union[HeadNode, SlurmQueue, LoginNodesPool],
9494
"ParallelClusterVersion": COOKBOOK_PACKAGES_VERSIONS["parallelcluster"],
9595
"CookbookVersion": COOKBOOK_PACKAGES_VERSIONS["cookbook"],
9696
"ChefVersion": COOKBOOK_PACKAGES_VERSIONS["chef"],
97-
"BerkshelfVersion": COOKBOOK_PACKAGES_VERSIONS["berkshelf"],
9897
}
9998

10099

‎cloudformation/external-slurmdbd/external-slurmdbd.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -434,7 +434,7 @@
434434
"UserData": {
435435
"Fn::Base64": {
436436
"Fn::Sub": [
437-
"Content-Type: multipart/mixed; boundary=\"==BOUNDARY==\"\nMIME-Version: 1.0\n\n--==BOUNDARY==\nContent-Type: text/cloud-config; charset=us-ascii\nMIME-Version: 1.0\n\npackage_update: false\npackage_upgrade: false\nrepo_upgrade: none\ndatasource_list: [ Ec2, None ]\n\n--==BOUNDARY==\nContent-Type: text/x-shellscript; charset=\"us-ascii\"\nMIME-Version: 1.0\n#!/bin/bash -x\n\nfunction vendor_cookbook\n{\n mkdir /tmp/cookbooks\n cd /tmp/cookbooks\n tar -xzf /etc/chef/aws-parallelcluster-cookbook.tgz\n HOME_BAK=\"${!HOME}\"\n export HOME=\"/tmp\"\n for d in /tmp/cookbooks/*; do\n cd \"$d\" || continue\n LANG=en_US.UTF-8 /opt/cinc/embedded/bin/berks vendor /etc/chef/cookbooks --delete\n done;\n export HOME=\"${!HOME_BAK}\"\n}\n\nfunction wait_for_private_ip_assignment\n{\n rc=1\n retries=10\n retry=1\n sleeptime=1\n while [ \\( $rc -eq 1 \\) -a \\( $retry -le $retries \\) ]; do\n number_of_ips=$(curl -H \"X-aws-ec2-metadata-token: $TOKEN\" -v http://169.254.169.254/latest/meta-data/network/interfaces/macs/\"$MAC\"/local-ipv4s | wc -l)\n ((number_of_ips>0))\n rc=$?\n retry=$((retry+1))\n sleep $sleeptime\n done\n return $rc\n}\n\nTOKEN=`curl -X PUT \"http://169.254.169.254/latest/api/token\" -H \"X-aws-ec2-metadata-token-ttl-seconds: 21600\"`\nMAC=$(curl -H \"X-aws-ec2-metadata-token: $TOKEN\" -v http://169.254.169.254/latest/meta-data/mac)\nENI_ID=$(curl -H \"X-aws-ec2-metadata-token: $TOKEN\" -v http://169.254.169.254/latest/meta-data/network/interfaces/macs/\"$MAC\"/interface-id)\nDEVICE_NAME=$(ls /sys/class/net | grep e)\n\n# Configure AWS CLI using the expected overrides, if any.\n[ -f /etc/profile.d/aws-cli-default-config.sh ] && . /etc/profile.d/aws-cli-default-config.sh\n\naws ec2 assign-private-ip-addresses --region \"${AWS::Region}\" --network-interface-id \"${!ENI_ID}\" --private-ip-addresses ${PrivateIp} --allow-reassignment\n\nwait_for_private_ip_assignment || echo \"Assignment of private IP ${PrivateIp} was not successful.\"\n\nip addr add ${PrivateIp}/${SubnetPrefix} dev \"${!DEVICE_NAME}\"\n\nif [ \"${CustomCookbookUrl}\" != \"NONE\" ]; then\n curl --retry 3 -v -L -o /etc/chef/aws-parallelcluster-cookbook.tgz ${CustomCookbookUrl}\n vendor_cookbook\nfi\n\n# This is necessary to find the cfn-init application\nexport PATH=/opt/aws/bin:${!PATH}\n[ -f /etc/parallelcluster/pcluster_cookbook_environment.sh ] && . /etc/parallelcluster/pcluster_cookbook_environment.sh\n\n$CFN_BOOTSTRAP_VIRTUALENV_PATH/cfn-init -s ${AWS::StackName} -v -c default -r LaunchTemplate --region \"${AWS::Region}\"\n",
437+
"Content-Type: multipart/mixed; boundary=\"==BOUNDARY==\"\nMIME-Version: 1.0\n\n--==BOUNDARY==\nContent-Type: text/cloud-config; charset=us-ascii\nMIME-Version: 1.0\n\npackage_update: false\npackage_upgrade: false\nrepo_upgrade: none\ndatasource_list: [ Ec2, None ]\n\n--==BOUNDARY==\nContent-Type: text/x-shellscript; charset=\"us-ascii\"\nMIME-Version: 1.0\n#!/bin/bash -x\n\nfunction vendor_cookbook\n{\n mkdir /tmp/cookbooks\n cd /tmp/cookbooks\n tar -xzf /etc/chef/aws-parallelcluster-cookbook.tgz --strip-components 1\n}\n\nfunction wait_for_private_ip_assignment\n{\n rc=1\n retries=10\n retry=1\n sleeptime=1\n while [ \\( $rc -eq 1 \\) -a \\( $retry -le $retries \\) ]; do\n number_of_ips=$(curl -H \"X-aws-ec2-metadata-token: $TOKEN\" -v http://169.254.169.254/latest/meta-data/network/interfaces/macs/\"$MAC\"/local-ipv4s | wc -l)\n ((number_of_ips>0))\n rc=$?\n retry=$((retry+1))\n sleep $sleeptime\n done\n return $rc\n}\n\nTOKEN=`curl -X PUT \"http://169.254.169.254/latest/api/token\" -H \"X-aws-ec2-metadata-token-ttl-seconds: 21600\"`\nMAC=$(curl -H \"X-aws-ec2-metadata-token: $TOKEN\" -v http://169.254.169.254/latest/meta-data/mac)\nENI_ID=$(curl -H \"X-aws-ec2-metadata-token: $TOKEN\" -v http://169.254.169.254/latest/meta-data/network/interfaces/macs/\"$MAC\"/interface-id)\nDEVICE_NAME=$(ls /sys/class/net | grep e)\n\n# Configure AWS CLI using the expected overrides, if any.\n[ -f /etc/profile.d/aws-cli-default-config.sh ] && . /etc/profile.d/aws-cli-default-config.sh\n\naws ec2 assign-private-ip-addresses --region \"${AWS::Region}\" --network-interface-id \"${!ENI_ID}\" --private-ip-addresses ${PrivateIp} --allow-reassignment\n\nwait_for_private_ip_assignment || echo \"Assignment of private IP ${PrivateIp} was not successful.\"\n\nip addr add ${PrivateIp}/${SubnetPrefix} dev \"${!DEVICE_NAME}\"\n\nif [ \"${CustomCookbookUrl}\" != \"NONE\" ]; then\n curl --retry 3 -v -L -o /etc/chef/aws-parallelcluster-cookbook.tgz ${CustomCookbookUrl}\n vendor_cookbook\nfi\n\n# This is necessary to find the cfn-init application\nexport PATH=/opt/aws/bin:${!PATH}\n[ -f /etc/parallelcluster/pcluster_cookbook_environment.sh ] && . /etc/parallelcluster/pcluster_cookbook_environment.sh\n\n$CFN_BOOTSTRAP_VIRTUALENV_PATH/cfn-init -s ${AWS::StackName} -v -c default -r LaunchTemplate --region \"${AWS::Region}\"\n",
438438
{
439439
"CustomCookbookUrl": {
440440
"Ref": "CustomCookbookUrl"

‎cloudformation/external-slurmdbd/resources/user_data.sh

Lines changed: 1 addition & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -19,14 +19,7 @@ function vendor_cookbook
1919
{
2020
mkdir /tmp/cookbooks
2121
cd /tmp/cookbooks
22-
tar -xzf /etc/chef/aws-parallelcluster-cookbook.tgz
23-
HOME_BAK="${!HOME}"
24-
export HOME="/tmp"
25-
for d in /tmp/cookbooks/*; do
26-
cd "$d" || continue
27-
LANG=en_US.UTF-8 /opt/cinc/embedded/bin/berks vendor /etc/chef/cookbooks --delete
28-
done;
29-
export HOME="${!HOME_BAK}"
22+
tar -xzf /etc/chef/aws-parallelcluster-cookbook.tgz --strip-components 1
3023
}
3124

3225
function wait_for_private_ip_assignment

0 commit comments

Comments
 (0)