Support for more than 1 Azure Storage account #4445
Description
New feature
Azure currently only supports a single storage account. In a few instances, there is a need to work across multiple storage accounts. I've had to do this once, I have been contacted a few times by other users who wish to do this.
Usage scenario
Typically this is for one of a few reasons:
- Authentication reasons (e.g. access control via storage accounts).
- Only keeping a single copy of some files and using them in multiple places (e.g. reference data).
- Ingress/egress strategy (e.g. machine from a sequencer is then processed in the analysis storage account).
Suggest implementation
The actual how to do this isn't as hard as how to expose this to the pipeline developer and user. Somehow, we have to delineate between different storage accounts that have the same path and name, e.g. az://data/R1.fastq could exist twice. Furthermore, isolating this to a single storage account provides certain benefits (e.g. no cross-region data transfer, simpler access control). Perhaps extending authentication methods such as #3314 will make workarounds easier (e.g. use azcopy to transfer data between accounts within a process).
Consider this more of a discussion on practicality rather than a serious feature.