Teardown - Failed to set ACL 'user::rwx,group::r-x,other::---' on container 'egress'. #2412
Description
✅ Checklist
- I have searched open and closed issues for duplicates.
- This is a problem observed when managing a Data Safe Haven.
- I can reproduce this with the latest version.
- I have read through the documentation.
- This isn't an open-ended question (open a discussion if it is).
💻 System information
- Operating System: MacOS
- Data Safe Haven version: 5.3.0
📦 Packages
List of packages
Paste list of packages here
🚫 Describe the problem
When trying to teardown an SRE deployed under 5.3.0 I get the following error:
pulumi:pulumi:Stack data-safe-haven-shm-prod5-sre-dsgadvicejan25 --- Logging error ---
pulumi:pulumi:Stack data-safe-haven-shm-prod5-sre-dsgadvicejan25 Traceback (most recent call last):
pulumi:pulumi:Stack data-safe-haven-shm-prod5-sre-dsgadvicejan25 File
"/Users/hduncan/.local/pipx/venvs/data-safe-haven/lib/python3.12/site-packages/azure/storage/filedatalake/_path_client.py",
line 726, in _set_access_control_internal
pulumi:pulumi:Stack data-safe-haven-shm-prod5-sre-dsgadvicejan25 headers, resp =
self._client.path.set_access_control_recursive(**options)
pulumi:pulumi:Stack data-safe-haven-shm-prod5-sre-dsgadvicejan25
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pulumi:pulumi:Stack data-safe-haven-shm-prod5-sre-dsgadvicejan25 File
"/Users/hduncan/.local/pipx/venvs/data-safe-haven/lib/python3.12/site-packages/azure/core/tracing/decorator.py", line 105, in
wrapper_use_tracer
pulumi:pulumi:Stack data-safe-haven-shm-prod5-sre-dsgadvicejan25 return func(*args, **kwargs)
pulumi:pulumi:Stack data-safe-haven-shm-prod5-sre-dsgadvicejan25 ^^^^^^^^^^^^^^^^^^^^^
pulumi:pulumi:Stack data-safe-haven-shm-prod5-sre-dsgadvicejan25 File
"/Users/hduncan/.local/pipx/venvs/data-safe-haven/lib/python3.12/site-packages/azure/storage/filedatalake/_generated/operation
s/_path_operations.py", line 2292, in set_access_control_recursive
pulumi:pulumi:Stack data-safe-haven-shm-prod5-sre-dsgadvicejan25 raise
HttpResponseError(response=response, model=error)
pulumi:pulumi:Stack data-safe-haven-shm-prod5-sre-dsgadvicejan25
azure.core.exceptions.HttpResponseError: Operation returned an invalid status 'Internal Server Error'
pulumi:pulumi:Stack data-safe-haven-shm-prod5-sre-dsgadvicejan25 During handling of the above
exception, another exception occurred:
pulumi:pulumi:Stack data-safe-haven-shm-prod5-sre-dsgadvicejan25 Traceback (most recent call last):
pulumi:pulumi:Stack data-safe-haven-shm-prod5-sre-dsgadvicejan25 File
"/opt/homebrew/Cellar/python@3.12/3.12.9/Frameworks/Python.framework/Versions/3.12/lib/python3.12/logging/__init__.py", line
1160, in emit
pulumi:pulumi:Stack data-safe-haven-shm-prod5-sre-dsgadvicejan25 msg = self.format(record)
pulumi:pulumi:Stack data-safe-haven-shm-prod5-sre-dsgadvicejan25 ^^^^^^^^^^^^^^^^^^^
pulumi:pulumi:Stack data-safe-haven-shm-prod5-sre-dsgadvicejan25 File
"/opt/homebrew/Cellar/python@3.12/3.12.9/Frameworks/Python.framework/Versions/3.12/lib/python3.12/logging/__init__.py", line
999, in format
pulumi:pulumi:Stack data-safe-haven-shm-prod5-sre-dsgadvicejan25 return fmt.format(record)
pulumi:pulumi:Stack data-safe-haven-shm-prod5-sre-dsgadvicejan25 ^^^^^^^^^^^^^^^^^^
pulumi:pulumi:Stack data-safe-haven-shm-prod5-sre-dsgadvicejan25 File
"/opt/homebrew/Cellar/python@3.12/3.12.9/Frameworks/Python.framework/Versions/3.12/lib/python3.12/logging/__init__.py", line
703, in format
pulumi:pulumi:Stack data-safe-haven-shm-prod5-sre-dsgadvicejan25 record.message =
record.getMessage()
pulumi:pulumi:Stack data-safe-haven-shm-prod5-sre-dsgadvicejan25
^^^^^^^^^^^^^^^^^^^
pulumi:pulumi:Stack data-safe-haven-shm-prod5-sre-dsgadvicejan25 File
"/opt/homebrew/Cellar/python@3.12/3.12.9/Frameworks/Python.framework/Versions/3.12/lib/python3.12/logging/__init__.py", line
392, in getMessage
pulumi:pulumi:Stack data-safe-haven-shm-prod5-sre-dsgadvicejan25 msg = msg % self.args
pulumi:pulumi:Stack data-safe-haven-shm-prod5-sre-dsgadvicejan25 ~~~~^~~~~~~~~~~
pulumi:pulumi:Stack data-safe-haven-shm-prod5-sre-dsgadvicejan25 TypeError: must be real number, not
type
pulumi:pulumi:Stack data-safe-haven-shm-prod5-sre-dsgadvicejan25 Call stack:
pulumi:pulumi:Stack data-safe-haven-shm-prod5-sre-dsgadvicejan25 File
"/opt/homebrew/Cellar/python@3.12/3.12.9/Frameworks/Python.framework/Versions/3.12/lib/python3.12/threading.py", line 1032, in
_bootstrap
pulumi:pulumi:Stack data-safe-haven-shm-prod5-sre-dsgadvicejan25 self._bootstrap_inner()
pulumi:pulumi:Stack data-safe-haven-shm-prod5-sre-dsgadvicejan25 File
"/opt/homebrew/Cellar/python@3.12/3.12.9/Frameworks/Python.framework/Versions/3.12/lib/python3.12/threading.py", line 1075, in
_bootstrap_inner
pulumi:pulumi:Stack data-safe-haven-shm-prod5-sre-dsgadvicejan25 self.run()
pulumi:pulumi:Stack data-safe-haven-shm-prod5-sre-dsgadvicejan25 File
"/opt/homebrew/Cellar/python@3.12/3.12.9/Frameworks/Python.framework/Versions/3.12/lib/python3.12/threading.py", line 1012, in
run
pulumi:pulumi:Stack data-safe-haven-shm-prod5-sre-dsgadvicejan25 self._target(*self._args,
**self._kwargs)
pulumi:pulumi:Stack data-safe-haven-shm-prod5-sre-dsgadvicejan25 File
"/opt/homebrew/Cellar/python@3.12/3.12.9/Frameworks/Python.framework/Versions/3.12/lib/python3.12/concurrent/futures/thread.py
", line 93, in _worker
pulumi:pulumi:Stack data-safe-haven-shm-prod5-sre-dsgadvicejan25 work_item.run()
pulumi:pulumi:Stack data-safe-haven-shm-prod5-sre-dsgadvicejan25 File
"/opt/homebrew/Cellar/python@3.12/3.12.9/Frameworks/Python.framework/Versions/3.12/lib/python3.12/concurrent/futures/thread.py
", line 59, in run
pulumi:pulumi:Stack data-safe-haven-shm-prod5-sre-dsgadvicejan25 result = self.fn(*self.args,
**self.kwargs)
pulumi:pulumi:Stack data-safe-haven-shm-prod5-sre-dsgadvicejan25 File
"/Users/hduncan/.local/pipx/venvs/data-safe-haven/lib/python3.12/site-packages/grpc/_server.py", line 793, in
_unary_response_in_pool
pulumi:pulumi:Stack data-safe-haven-shm-prod5-sre-dsgadvicejan25 response, proceed =
_call_behavior(
pulumi:pulumi:Stack data-safe-haven-shm-prod5-sre-dsgadvicejan25 File
"/Users/hduncan/.local/pipx/venvs/data-safe-haven/lib/python3.12/site-packages/grpc/_server.py", line 610, in _call_behavior
pulumi:pulumi:Stack data-safe-haven-shm-prod5-sre-dsgadvicejan25 response_or_iterator =
behavior(argument, context)
pulumi:pulumi:Stack data-safe-haven-shm-prod5-sre-dsgadvicejan25 File
"/Users/hduncan/.local/pipx/venvs/data-safe-haven/lib/python3.12/site-packages/pulumi/dynamic/__main__.py", line 149, in
Delete
pulumi:pulumi:Stack data-safe-haven-shm-prod5-sre-dsgadvicejan25 provider.delete(id_, props)
pulumi:pulumi:Stack data-safe-haven-shm-prod5-sre-dsgadvicejan25 File
"/Users/hduncan/.local/pipx/venvs/data-safe-haven/lib/python3.12/site-packages/data_safe_haven/infrastructure/components/dynam
ic/blob_container_acl.py", line 79, in delete
pulumi:pulumi:Stack data-safe-haven-shm-prod5-sre-dsgadvicejan25
azure_sdk.set_blob_container_acl(
pulumi:pulumi:Stack data-safe-haven-shm-prod5-sre-dsgadvicejan25 File
"/Users/hduncan/.local/pipx/venvs/data-safe-haven/lib/python3.12/site-packages/data_safe_haven/external/api/azure_sdk.py",
line 1300, in set_blob_container_acl
pulumi:pulumi:Stack data-safe-haven-shm-prod5-sre-dsgadvicejan25
directory_client.set_access_control_recursive(acl=desired_acl)
pulumi:pulumi:Stack data-safe-haven-shm-prod5-sre-dsgadvicejan25 File
"/Users/hduncan/.local/pipx/venvs/data-safe-haven/lib/python3.12/site-packages/azure/core/tracing/decorator.py", line 105, in
wrapper_use_tracer
pulumi:pulumi:Stack data-safe-haven-shm-prod5-sre-dsgadvicejan25 return func(*args, **kwargs)
pulumi:pulumi:Stack data-safe-haven-shm-prod5-sre-dsgadvicejan25 File
"/Users/hduncan/.local/pipx/venvs/data-safe-haven/lib/python3.12/site-packages/azure/storage/filedatalake/_path_client.py",
line 605, in set_access_control_recursive
pulumi:pulumi:Stack data-safe-haven-shm-prod5-sre-dsgadvicejan25 return
self._set_access_control_internal(options=options, progress_hook=progress_hook,
pulumi:pulumi:Stack data-safe-haven-shm-prod5-sre-dsgadvicejan25 File
"/Users/hduncan/.local/pipx/venvs/data-safe-haven/lib/python3.12/site-packages/azure/storage/filedatalake/_path_client.py",
line 772, in _set_access_control_internal
pulumi:pulumi:Stack data-safe-haven-shm-prod5-sre-dsgadvicejan25 process_storage_error(error)
pulumi:pulumi:Stack data-safe-haven-shm-prod5-sre-dsgadvicejan25 File
"/Users/hduncan/.local/pipx/venvs/data-safe-haven/lib/python3.12/site-packages/azure/storage/filedatalake/_deserialize.py",
line 166, in process_storage_error
pulumi:pulumi:Stack data-safe-haven-shm-prod5-sre-dsgadvicejan25 _LOGGER.warning(
pulumi:pulumi:Stack data-safe-haven-shm-prod5-sre-dsgadvicejan25 Message: 'Unexpected return type %
from ContentDecodePolicy.deserialize_from_http_generics.'
pulumi:pulumi:Stack data-safe-haven-shm-prod5-sre-dsgadvicejan25 Arguments: (<class 'NoneType'>,)
pulumi:pulumi:Stack data-safe-haven-shm-prod5-sre-dsgadvicejan25 Failed to set ACL
'user::rwx,group::r-x,other::---' on container 'egress'.
pulumi:pulumi:Stack data-safe-haven-shm-prod5-sre-dsgadvicejan25 Failed to delete custom ACLs on
storage account 'shprosredsgsensitivedata'.
- pulumi-python:dynamic:Resource sre_data_blob_egress_blob_container_egress_acl deleting (326s)
error: Exception calling application: Failed to delete custom ACLs on storage account
'shprosredsgsensitivedata'.
- pulumi-python:dynamic:Resource sre_data_blob_egress_blob_container_egress_acl **deleting failed**
error: Exception calling application: Failed to delete custom ACLs on storage account
'shprosredsgsensitivedata'.
@ destroying......
pulumi:pulumi:Stack data-safe-haven-shm-prod5-sre-dsgadvicejan25 error: update failed
pulumi:pulumi:Stack data-safe-haven-shm-prod5-sre-dsgadvicejan25 **failed** 1 error; 65
messages
Diagnostics:
pulumi-python:dynamic:Resource (sre_data_blob_egress_blob_container_egress_acl):
error: Exception calling application: Failed to delete custom ACLs on storage account
'shprosredsgsensitivedata'.
pulumi:pulumi:Stack (data-safe-haven-shm-prod5-sre-dsgadvicejan25):
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1740395312.979310 6183956 fork_posix.cc:77] Other threads are currently calling into gRPC, skipping
fork() handlers
I0000 00:00:1740395313.034001 6183956 fork_posix.cc:77] Other threads are currently calling into gRPC, skipping
fork() handlers
I0000 00:00:1740395316.563066 6183956 fork_posix.cc:77] Other threads are currently calling into gRPC, skipping
fork() handlers
I0000 00:00:1740395318.498553 6183956 fork_posix.cc:77] Other threads are currently calling into gRPC, skipping
fork() handlers
I0000 00:00:1740395319.480276 6183956 fork_posix.cc:77] Other threads are currently calling into gRPC, skipping
fork() handlers
--- Logging error ---
Traceback (most recent call last):
File
"/Users/hduncan/.local/pipx/venvs/data-safe-haven/lib/python3.12/site-packages/azure/storage/filedatalake/_path_client.py",
line 726, in _set_access_control_internal
headers, resp = self._client.path.set_access_control_recursive(**options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/Users/hduncan/.local/pipx/venvs/data-safe-haven/lib/python3.12/site-packages/azure/core/tracing/decorator.py", line 105, in
wrapper_use_tracer
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File
"/Users/hduncan/.local/pipx/venvs/data-safe-haven/lib/python3.12/site-packages/azure/storage/filedatalake/_generated/operation
s/_path_operations.py", line 2292, in set_access_control_recursive
raise HttpResponseError(response=response, model=error)
azure.core.exceptions.HttpResponseError: Operation returned an invalid status 'Internal Server Error'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File
"/opt/homebrew/Cellar/python@3.12/3.12.9/Frameworks/Python.framework/Versions/3.12/lib/python3.12/logging/__init__.py", line
1160, in emit
msg = self.format(record)
^^^^^^^^^^^^^^^^^^^
File
"/opt/homebrew/Cellar/python@3.12/3.12.9/Frameworks/Python.framework/Versions/3.12/lib/python3.12/logging/__init__.py", line
999, in format
return fmt.format(record)
^^^^^^^^^^^^^^^^^^
File
"/opt/homebrew/Cellar/python@3.12/3.12.9/Frameworks/Python.framework/Versions/3.12/lib/python3.12/logging/__init__.py", line
703, in format
record.message = record.getMessage()
^^^^^^^^^^^^^^^^^^^
File
"/opt/homebrew/Cellar/python@3.12/3.12.9/Frameworks/Python.framework/Versions/3.12/lib/python3.12/logging/__init__.py", line
392, in getMessage
msg = msg % self.args
~~~~^~~~~~~~~~~
TypeError: must be real number, not type
Call stack:
File
"/opt/homebrew/Cellar/python@3.12/3.12.9/Frameworks/Python.framework/Versions/3.12/lib/python3.12/threading.py", line 1032, in
_bootstrap
self._bootstrap_inner()
File
"/opt/homebrew/Cellar/python@3.12/3.12.9/Frameworks/Python.framework/Versions/3.12/lib/python3.12/threading.py", line 1075, in
_bootstrap_inner
self.run()
File
"/opt/homebrew/Cellar/python@3.12/3.12.9/Frameworks/Python.framework/Versions/3.12/lib/python3.12/threading.py", line 1012, in
run
self._target(*self._args, **self._kwargs)
File
"/opt/homebrew/Cellar/python@3.12/3.12.9/Frameworks/Python.framework/Versions/3.12/lib/python3.12/concurrent/futures/thread.py
", line 93, in _worker
work_item.run()
File
"/opt/homebrew/Cellar/python@3.12/3.12.9/Frameworks/Python.framework/Versions/3.12/lib/python3.12/concurrent/futures/thread.py
", line 59, in run
result = self.fn(*self.args, **self.kwargs)
File "/Users/hduncan/.local/pipx/venvs/data-safe-haven/lib/python3.12/site-packages/grpc/_server.py", line 793, in
_unary_response_in_pool
response, proceed = _call_behavior(
File "/Users/hduncan/.local/pipx/venvs/data-safe-haven/lib/python3.12/site-packages/grpc/_server.py", line 610, in
_call_behavior
response_or_iterator = behavior(argument, context)
File "/Users/hduncan/.local/pipx/venvs/data-safe-haven/lib/python3.12/site-packages/pulumi/dynamic/__main__.py",
line 149, in Delete
provider.delete(id_, props)
File
"/Users/hduncan/.local/pipx/venvs/data-safe-haven/lib/python3.12/site-packages/data_safe_haven/infrastructure/components/dynam
ic/blob_container_acl.py", line 79, in delete
azure_sdk.set_blob_container_acl(
File
"/Users/hduncan/.local/pipx/venvs/data-safe-haven/lib/python3.12/site-packages/data_safe_haven/external/api/azure_sdk.py",
line 1300, in set_blob_container_acl
directory_client.set_access_control_recursive(acl=desired_acl)
File
"/Users/hduncan/.local/pipx/venvs/data-safe-haven/lib/python3.12/site-packages/azure/core/tracing/decorator.py", line 105, in
wrapper_use_tracer
return func(*args, **kwargs)
File
"/Users/hduncan/.local/pipx/venvs/data-safe-haven/lib/python3.12/site-packages/azure/storage/filedatalake/_path_client.py",
line 605, in set_access_control_recursive
return self._set_access_control_internal(options=options, progress_hook=progress_hook,
File
"/Users/hduncan/.local/pipx/venvs/data-safe-haven/lib/python3.12/site-packages/azure/storage/filedatalake/_path_client.py",
line 772, in _set_access_control_internal
process_storage_error(error)
File
"/Users/hduncan/.local/pipx/venvs/data-safe-haven/lib/python3.12/site-packages/azure/storage/filedatalake/_deserialize.py",
line 166, in process_storage_error
_LOGGER.warning(
Message: 'Unexpected return type % from ContentDecodePolicy.deserialize_from_http_generics.'
Arguments: (<class 'NoneType'>,)
Failed to set ACL 'user::rwx,group::r-x,other::---' on container 'egress'.
Failed to delete custom ACLs on storage account 'shprosredsgsensitivedata'.
error: update failed
Resources:
Duration: 5m31s
Pulumi error: - pulumi-python:dynamic:Resource sre_data_blob_egress_blob_container_egress_acl
deleting (326s) error: Exception calling application: Failed to delete custom ACLs on storage
account 'shprosredsgsensitivedata'.
Pulumi error: - pulumi-python:dynamic:Resource sre_data_blob_egress_blob_container_egress_acl **deleting
failed** error: Exception calling application: Failed to delete custom ACLs on storage account
'shprosredsgsensitivedata'.
Pulumi error: pulumi:pulumi:Stack data-safe-haven-shm-prod5-sre-dsgadvicejan25 error:
update failed
Pulumi error: error: Exception calling application: Failed to delete custom ACLs on storage account
'shprosredsgsensitivedata'.
Pulumi error: error: update failed
Pulumi error: stderr:
Pulumi resource destruction failed.
Pulumi destroy failed.
Tearing down Pulumi infrastructure failed..
Could not teardown Secure Research Environment 'dsgadvicejan25'.
- I have updated Pulumi
- I have checked on portal that the egress storage container and network are as described in the config
- I have tried 3 times to teardown