Skip to content

Add EFS access point support (aws-parallelcluster#2337) #6359

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 2 commits into
base: develop
Choose a base branch
from
Draft

Add EFS access point support (aws-parallelcluster#2337) #6359

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
9b6d7a4
Add EFS access point support (aws-parallelcluster#2337)
thheinen Jul 19, 2024
a752e2e
Rename new setting, Add changelog, Fix style, fix some tests
thheinen Aug 2, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions cli/src/pcluster/config/cluster_config.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -371,6 +371,7 @@ def __init__(
deletion_policy: str = None,
encryption_in_transit: bool = None,
iam_authorization: bool = None,
accesspoint_id: str = None,
):
super().__init__()
self.mount_dir = Resource.init_param(mount_dir)
Expand All @@ -387,6 +388,7 @@ def __init__(
)
self.encryption_in_transit = Resource.init_param(encryption_in_transit, default=False)
self.iam_authorization = Resource.init_param(iam_authorization, default=False)
self.accesspoint_id = Resource.init_param(accesspoint_id)

def _register_validators(self, context: ValidatorContext = None): # noqa: D102 #pylint: disable=unused-argument
self._register_validator(SharedStorageNameValidator, name=self.name)
Expand All @@ -398,6 +400,7 @@ def _register_validators(self, context: ValidatorContext = None): # noqa: D102
EfsMountOptionsValidator,
encryption_in_transit=self.encryption_in_transit,
iam_authorization=self.iam_authorization,
accesspoint_id=self.accesspoint_id,
name=self.name,
)

Expand Down
5 changes: 4 additions & 1 deletion cli/src/pcluster/schemas/cluster_schema.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -321,6 +321,9 @@ class EfsSettingsSchema(BaseSchema):
deletion_policy = fields.Str(
validate=validate.OneOf(DELETION_POLICIES), metadata={"update_policy": UpdatePolicy.SUPPORTED}
)
accesspoint_id = fields.Str(
validate=validate.Regexp(r"^fsap-[0-9a-z]{17}$"),
)
encryption_in_transit = fields.Bool(metadata={"update_policy": UpdatePolicy.UNSUPPORTED})
iam_authorization = fields.Bool(metadata={"update_policy": UpdatePolicy.UNSUPPORTED})

Expand All @@ -331,7 +334,7 @@ def validate_file_system_id_ignored_parameters(self, data, **kwargs):
messages = []
if data.get("file_system_id") is not None:
for key in data:
if key is not None and key not in ["encryption_in_transit", "iam_authorization", "file_system_id"]:
if key is not None and key not in ["encryption_in_transit", "iam_authorization", "file_system_id", "accesspoint_id"]:
messages.append(EFS_MESSAGES["errors"]["ignored_param_with_efs_fs_id"].format(efs_param=key))
if messages:
raise ValidationError(message=messages)
Expand Down
5 changes: 5 additions & 0 deletions cli/src/pcluster/templates/cluster_stack.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1109,6 +1109,7 @@ def _add_efs_storage(self, id: str, shared_efs: SharedEfs):
shared_efs.encryption_in_transit
)
self.shared_storage_attributes[SharedStorageType.EFS]["IamAuthorizations"].append(shared_efs.iam_authorization)
self.shared_storage_attributes[SharedStorageType.EFS]["AccesspointIds"].append(shared_efs.accesspoint_id)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here and in other files, we are assuming that more than one access point can be specified for a given file system (AccesspointsIds rather than AccesspointId).

It's true that a file system can support multiple access points (https://docs.aws.amazon.com/efs/latest/ug/efs-access-points.html#efs-access-poiont-create), but the mount action is always done using a single access point (https://docs.aws.amazon.com/efs/latest/ug/mounting-access-points.html).

So please, let's assume to specify only one access point on the cluster config.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is actually a per filesystem array, similar to the encryption in transit array

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it's actually an array of all the separate FS in config. I just aligned naming and structure to the other properties which are also named in plural.


return efs_id

Expand Down Expand Up @@ -1288,6 +1289,10 @@ def _add_head_node(self):
"efs_iam_authorizations": to_comma_separated_string(
self.shared_storage_attributes[SharedStorageType.EFS]["IamAuthorizations"], use_lower_case=True
),
"efs_accesspoint_ids": to_comma_separated_string(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change should be captured in unit test here: https://github.com/aws/aws-parallelcluster/blob/develop/cli/tests/pcluster/templates/test_cluster_stack.py#L901

self.shared_storage_attributes[SharedStorageType.EFS]["AccesspointIds"],
use_lower_case=True,
),
"fsx_fs_ids": get_shared_storage_ids_by_type(self.shared_storage_infos, SharedStorageType.FSX),
"fsx_mount_names": to_comma_separated_string(
self.shared_storage_attributes[SharedStorageType.FSX]["MountNames"]
Expand Down
4 changes: 4 additions & 0 deletions cli/src/pcluster/templates/login_nodes_stack.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -235,6 +235,10 @@ def _add_login_nodes_pool_launch_template(self):
self._shared_storage_attributes[SharedStorageType.EFS]["IamAuthorizations"],
use_lower_case=True,
),
"efs_accesspoint_ids": to_comma_separated_string(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change should be captured in unit test here: https://github.com/aws/aws-parallelcluster/blob/develop/cli/tests/pcluster/templates/test_login_nodes_stack.py#L23

self._shared_storage_attributes[SharedStorageType.EFS]["AccesspointIds"],
use_lower_case=True,
),
"enable_intel_hpc_platform": "true" if self._config.is_intel_hpc_platform_enabled else "false",
"ephemeral_dir": DEFAULT_EPHEMERAL_DIR,
"fsx_fs_ids": get_shared_storage_ids_by_type(self._shared_storage_infos, SharedStorageType.FSX),
Expand Down
4 changes: 4 additions & 0 deletions cli/src/pcluster/templates/queues_stack.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -335,6 +335,10 @@ def _add_compute_resource_launch_template(
self._shared_storage_attributes[SharedStorageType.EFS]["IamAuthorizations"],
use_lower_case=True,
),
"efs_accesspoint_ids": to_comma_separated_string(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change should be captured in unit test here: https://github.com/aws/aws-parallelcluster/blob/develop/cli/tests/pcluster/templates/test_queues_stack.py#L114

self._shared_storage_attributes[SharedStorageType.EFS]["AccesspointIds"],
use_lower_case=True,
),
"fsx_fs_ids": get_shared_storage_ids_by_type(self._shared_storage_infos, SharedStorageType.FSX),
"fsx_mount_names": to_comma_separated_string(
self._shared_storage_attributes[SharedStorageType.FSX]["MountNames"]
Expand Down
2 changes: 1 addition & 1 deletion cli/src/pcluster/validators/efs_validators.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ class EfsMountOptionsValidator(Validator):
IAM Authorization requires Encryption in Transit.
"""

def _validate(self, encryption_in_transit: bool, iam_authorization: bool, name: str):
def _validate(self, encryption_in_transit: bool, iam_authorization: bool, accesspoint_id: str, name: str):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change should be captured in unit test here: https://github.com/aws/aws-parallelcluster/blob/develop/cli/tests/pcluster/validators/test_efs_validators.py

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The validator should verify that the provided access point exists

if iam_authorization and not encryption_in_transit:
self._add_failure(
"EFS IAM authorization cannot be enabled when encryption in-transit is disabled. "
Expand Down
1 change: 1 addition & 0 deletions cli/src/pcluster3_config_converter/pcluster3_config_converter.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -296,6 +296,7 @@ def convert_efs_settings(self, section_name):
("efs_kms_key_id", "KmsKeyId"),
("provisioned_throughput", "ProvisionedThroughput", "getint"),
("throughput_mode", "ThroughputMode"),
("accesspoint_id", "AccesspointId"),
]
efs_section, efs_dict, _section_label = self.convert_storage_base(
"efs", efs_label.strip(), additional_items
Expand Down
Loading