4.0.0 release notes

What is new?

• TES, which was previously part of CoA, is now a separate product that CoA will continue to utilize as a core component (see TES 4.0.0 release notes)
• Cosmos DB used by TES, and MySQL used by Cromwell, were both replaced by PostgreSQL to unify deployments on a single database platform
• The Azure Batch implementation was modified to optimize Azure Batch pool usage and increase scalability
• To determine the VM size to use, TES now uses the Azure Retail Prices API instead of the Azure RateCard API

What is fixed?

• Ability to pass containers-to-mount for AKS deployments in addition for VM deployments allowing users to mount additional containers outside the default ones deployed by CoA
• Updated Cromwell to Version 85 to solve the Disks/Disk differences in Runtime and make workflows more reproducible across clouds
• Bug fixes for storage accounts access
• LRS storage accounts created instead of GRS for new deployments to reduce cost
• Provide users an option to use existing managed access identity instead of insisting on creating a new one and erroring out if the managed access identity with the same name was present
• Test WDL's specifies docker images from MCR instead of ubuntu docker hub, reducing docker hub traffic, enabling faster pulls and fewer network errors in Azure

Upgrade Instructions

• Upgrading to version 4.0 and greater is not available for any previous deployment, but reuse of the current deployment’s resources is possible.
• Link to the Upgrade Guide: CromwellOnAzure/3.2-to-4.0.md at develop · microsoft/CromwellOnAzure (github.com)

Change Log

• Update README to indicate main and develop branches by @MattMcL4475 in https://github.com/microsoft/CromwellOnAzure/pull/433
• Update CromwellOnAzureVersion in env-00-coa-version.txt to 3.2.0 by @MattMcL4475 in https://github.com/microsoft/CromwellOnAzure/pull/434
• Run apt upgrade on Docker images on build, set all Dockerfiles to .NET 6, add CredScanSuppressions.json by @MattMcL4475 in https://github.com/microsoft/CromwellOnAzure/pull/442
• Cherry-pick commit to update Docker image base to .NET 6, upgrade on build, add CredScanSuppressions.json by @MattMcL4475 in https://github.com/microsoft/CromwellOnAzure/pull/446
• Replace obsolete API calls. See https://github.com/microsoft/ApplicationInsights-dotnet/issues/2560 for more details. by @BMurri in https://github.com/microsoft/CromwellOnAzure/pull/445
• Fix async/await exception handling bug in GetExistingStorageAccountAsync by @jsaun in https://github.com/microsoft/CromwellOnAzure/pull/449
• Create Azure storage account as type LRS by default instead of GRS by @MattMcL4475 in https://github.com/microsoft/CromwellOnAzure/pull/427
• Only generate and upload timing.html when workflow reaches terminal state by @MattMcL4475 in https://github.com/microsoft/CromwellOnAzure/pull/462
• Update TES OpenAPI info by @MattMcL4475 in https://github.com/microsoft/CromwellOnAzure/pull/465
• Fix blob listing when hierarchical namespace enabled on the storage account by @jsaun in https://github.com/microsoft/CromwellOnAzure/pull/464
• Switch to the MCR-hosted Ubuntu:22.04 image for test.wdl by @vsmalladi in https://github.com/microsoft/CromwellOnAzure/pull/470
• Fixed the issue where the system would become unusable if Batch account contained a pool of VM size that is not on the supported list by @tonybendis in https://github.com/microsoft/CromwellOnAzure/pull/472
• Check if managed identity exists before attempting to create it by @jsaun in https://github.com/microsoft/CromwellOnAzure/pull/473
• CromwellOnAzure On AKS by @jsaun in https://github.com/microsoft/CromwellOnAzure/pull/354
• Fix race condition in CreateBatchJobAsync logic by @MattMcL4475 in https://github.com/microsoft/CromwellOnAzure/pull/483
• Support managedId or keyvault auth default containers by @jsaun in https://github.com/microsoft/CromwellOnAzure/pull/484
• Fix visible text in deployer by @BMurri in https://github.com/microsoft/CromwellOnAzure/pull/486
• Bug fix: support specifying an existing CosmosDB account by @jsaun in https://github.com/microsoft/CromwellOnAzure/pull/491
• Refreshed the list of VM sizes and default prices. Added support for family names in allowed-vm-sizes config file. by @tonybendis in https://github.com/microsoft/CromwellOnAzure/pull/488
• Addressed several things in the deployer WRT AKS upgrades & others by @BMurri in https://github.com/microsoft/CromwellOnAzure/pull/489
• Reduce deployer HTTP timeout and handle exceptions in AssignVmAsBillingReaderToSubscriptionAsync by @MattMcL4475 in https://github.com/microsoft/CromwellOnAzure/pull/495
• Update Cromwell version to 84 by @MattMcL4475 in https://github.com/microsoft/CromwellOnAzure/pull/499
• Fix NullReferenceException in VM upgrades by @BMurri in https://github.com/microsoft/CromwellOnAzure/pull/504
• Move .net urls parameter to environment variable by @jsaun in https://github.com/microsoft/CromwellOnAzure/pull/503
• Remove Microsoft.Extensions.Logging.Abstractions by @MattMcL4475 in https://github.com/microsoft/CromwellOnAzure/pull/508
• Improve update reliability by @BMurri in https://github.com/microsoft/CromwellOnAzure/pull/513
• Add ability to pass start task as a env var to Tes by @jsaun in https://github.com/microsoft/CromwellOnAzure/pull/500
• Add securityContext and resourceLimits and apparmor to helm chart by @jsaun in https://github.com/microsoft/CromwellOnAzure/pull/502
• Misc aks followup by @jsaun in https://github.com/microsoft/CromwellOnAzure/pull/507
• Update AKS documentation by @jsaun in https://github.com/microsoft/CromwellOnAzure/pull/514
• Generate MYSQL_ROOT_PASSWORD by @BMurri in https://github.com/microsoft/CromwellOnAzure/pull/510
• Added information about integer values for preemptible by @olesya13 in https://github.com/microsoft/CromwellOnAzure/pull/509
• Fix start-task for VM case by @BMurri in https://github.com/microsoft/CromwellOnAzure/pull/516
• Added document on MySQL update by @olesya13 in https://github.com/microsoft/CromwellOnAzure/pull/506
• Fix Trigger Service test that occasionally fails on threading issue by @MattMcL4475 in https://github.com/microsoft/CromwellOnAzure/pull/521
• Set ContainerConfiguration when using Manual Pools by @MattMcL4475 in https://github.com/microsoft/CromwellOnAzure/pull/520
• Update all Nuget packages to latest by @MattMcL4475 in https://github.com/microsoft/CromwellOnAzure/pull/526
• Delete manual Batch pools regardless of presence of identity, and add more logging by @MattMcL4475 in https://github.com/microsoft/CromwellOnAzure/pull/525
• Misc. fixes for deploying with AKS by @MattMcL4475 in https://github.com/microsoft/CromwellOnAzure/pull/528
• Update default Helm installation location on Linux by @MattMcL4475 in https://github.com/microsoft/CromwellOnAzure/pull/540
• Address failure to run bash-based start task script by @BMurri in https://github.com/microsoft/CromwellOnAzure/pull/546
• Add 3.2.0 release notes by @MattMcL4475 in https://github.com/microsoft/CromwellOnAzure/pull/435
• Merge TES dockerfile by @MattMcL4475 in https://github.com/microsoft/CromwellOnAzure/pull/548
• Add GA4GH TES as submodule and remove TES code by @MattMcL4475 in https://github.com/microsoft/CromwellOnAzure/pull/551
• Update CoA to 3.3, .NET to 7, NuGet packages to latest by @MattMcL4475 in https://github.com/microsoft/CromwellOnAzure/pull/559
• Exclude blobfuse paths from mlocate scanning by @jsaun in https://github.com/microsoft/CromwellOnAzure/pull/562
• Refactor for DI, fix whitespace and other fixes by @BMurri in https://github.com/microsoft/CromwellOnAzure/pull/564
• Change Azure Batch image to use HPC image by @MattMcL4475 in https://github.com/microsoft/CromwellOnAzure/pull/560
• Updates Tes version and CoA deployer to use the name env variable names for cosmos and batch account names. by @giventocode in https://github.com/microsoft/CromwellOnAzure/pull/573
• Refactor for DI, fix whitespace and other fixes by @BMurri in https://github.com/microsoft/CromwellOnAzure/pull/568
• Implement Shared Pools by @BMurri in https://github.com/microsoft/CromwellOnAzure/pull/574
• Align repos by @BMurri in https://github.com/microsoft/CromwellOnAzure/pull/587
• Implement support for TES to use PostgreSql by @ashanhol in https://github.com/microsoft/CromwellOnAzure/pull/567
• SharedPool improvements by @BMurri in https://github.com/microsoft/CromwellOnAzure/pull/586
• Update Cromwell to version 85 and update submodule by @MattMcL4475 in https://github.com/microsoft/CromwellOnAzure/pull/589
• Manual helm fixes by @jsaun in https://github.com/microsoft/CromwellOnAzure/pull/590
• Add ability to pass containers-to-mount at deployment time for AKS by @jsaun in https://github.com/microsoft/CromwellOnAzure/pull/584
• Add document for migrating to 4.0 by @jsaun in https://github.com/microsoft/CromwellOnAzure/pull/594
• Misc deployer fixes by @jsaun in https://github.com/microsoft/CromwellOnAzure/pull/593
• Update Dependencies by @BMurri in https://github.com/microsoft/CromwellOnAzure/pull/596
• Update CredScanSuppressions.json by @BMurri in https://github.com/microsoft/CromwellOnAzure/pull/599
• Migrate TES configuration to IOptions by @BMurri in https://github.com/microsoft/CromwellOnAzure/pull/592
• Missing changes and update submodule by @BMurri in https://github.com/microsoft/CromwellOnAzure/pull/600

New Contributors

• @giventocode made their first contribution in https://github.com/microsoft/CromwellOnAzure/pull/573

Full Changelog: https://github.com/microsoft/CromwellOnAzure/compare/3.2.0...4.0.0