53

53 Release Notes

Martha v3 Support

Cromwell now supports resolving DRS URIs through Martha v3 (in addition to Martha v2). To switch to the new version of Martha, update the martha.url found in the filesystems config to
point to /martha_v3. More information on Martha v3 request and response schema can be found here.

Support for custom entrypoints on Docker images

Cromwell can now support docker images which have custom entrypoints in the PAPIv2 alpha and beta backends.

Alpha support for WDL optional outputs on PAPI v2

• Alpha support for WDL optional output files on the PAPI v2 backend has been added, please see the
  documentation
  for known limitations.

Monitoring Image Script

• Cromwell now supports an optional monitoring_image_script workflow option in addition to the existing
  monitoring_script and monitoring_image options. For more information see the Google Pipelines API Workflow Options
  documentation.

52

52 Release Notes

Documentation

Information on how to properly use the Singularity cache with Cromwell is now
provided in the Cromwell Singularity documentation.

Google library upgrade (#5565)

All previous versions of Cromwell shipped with Google Cloud Storage (GCS) libraries that are now deprecated and will stop working in August 2020. This release adopts updated libraries to ensure uninterrupted operation. The only user action required is upgrading Cromwell.

Bug fixes

• Fixed a bug that required Cromwell to be restarted in order to pick up DNS changes.
  • By default, the JVM caches DNS records with a TTL of infinity.
  • Cromwell now configures its JVM with a 3-minute TTL. This value can be customized by setting system.dns-cache-ttl.
• Clarified an error message that Cromwell emits when the compute backend terminates a job of its own volition (as opposed to termination in response to an abort request from Cromwell)
  • Previously, the error read The job was aborted from outside Cromwell
  • The new error reads The compute backend terminated the job. If this termination is unexpected, examine likely causes such as preemption, running out of disk or memory on the compute instance, or exceeding the backend's maximum job duration.

51

51 Release Notes

Changes and Warnings

The configuration format for call cache blacklisting has been updated, please see the call caching documentation for details.

Bug fixes

• Fixed a bug where the size(...) function did not work correctly on files
  from a shared filesystem if size(...) was called in the input section on a
  relative path.

• Fixed a bug where the use_relative_output_paths option would not preserve intermediate folders.

New functionality

Call caching blacklisting improvements

Cromwell previously supported blacklisting GCS buckets containing cache hits which could not be copied for permissions
reasons. Cromwell now adds support for blacklisting individual cache hits which could not be copied for any reason,
as well as grouping blacklist caches according to a workflow option key. More information available in the
call caching documentation.

new xxh64 and fingerprint strategies for call caching

Existing call cache strategies path and path+modtime don't work when using docker on shared filesystems
(SFS backend, i.e. not in cloud storage). The file (md5sum) strategy works, but uses a lot of resources.
Two faster strategies have been added for this use case: xxh64 and
fingerprint. xxh64 is a lightweight hashing algorithm, fingerprint is a strategy designed to be very
lightweight. Read more about it in the call caching documentation.

50

50 Release Notes

Changes and Warnings

Metadata Archival Config Change

Note: Unless you have already opted-in to GCS-archival of metadata during its development, this change will not affect you.
Cromwell's metadata archival configuration has changed in a backwards incompatible way to increase consistency,
please see
the updated documentation for details.

49

49 Release Notes

Changes and Warnings

Job store database refactoring

The primary keys of Cromwell's job store tables have been refactored to use a BIGINT datatype in place of the previous
INT datatype. Cromwell will not be usable during the time the Liquibase migration for this refactor is running.
In the Google Cloud SQL with SSD environment this migration runs at a rate of approximately 40,000 JOB_STORE_SIMPLETON_ENTRY
rows per second. In deployments with millions or billions of JOB_STORE_SIMPLETON_ENTRY rows the migration may require
a significant amount of downtime so please plan accordingly. The following SQL could be used to estimate the number of
rows in this table:

SELECT table_rows FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_SCHEMA = 'cromwell' AND table_name = 'JOB_STORE_SIMPLETON_ENTRY';

Execution Directory Layout (cache copies)

When an attempt to copy a cache result is made, you'll now see a cacheCopy directory in the call root directory.
This prevents them clashing with the files staged to the same directory for attempt 1 if the cache copy fails (see also: Bug Fixes).

The directory layout used to be:

[...]/callRoot/
  - script [from the cache copy attempt, or for execution attempt 1 if the cache copy fails]
  - stdout [from the cache copy attempt, or for execution attempt 1 if the cache copy fails]
  - output.file [from the cache copy attempt, or for execution attempt 1 if the cache copy fails]
  - attempt-2/ [if attempt 1 fails]
    - script
    - stdout
    - output.file

but is now:

[...]/callRoot/
  - cacheCopy/
    - script
    - stdout
    - output.file
  - script [for attempt 1 if the cache copy fails]
  - stdout [for attempt 1 if the cache copy fails]
  - output.file [for attempt 1 if the cache copy fails]
  - attempt-2/ [if attempt 1 fails]
    - script
    - stdout
    - output.file

New Functionality

Disable call-caching for tasks

It is now possible to indicate in a workflow that a task should not be call-cached. See details
here.

Delete Intermediate Outputs on PapiV2

• Experimental: When a new workflow option delete_intermediate_output_files is submitted with the workflow,
  intermediate File objects will be deleted when the workflow completes. See the Google Pipelines API Workflow Options
  documentation
  for more information.

Metadata Archival Support

Cromwell 49 now offers the option to archive metadata to GCS and remove the equivalent metadata from relational
database storage. Please see
the documentation for more details.

Adding support for Google Cloud Life Sciences v2beta

Cromwell now supports running workflows using Google Cloud Life Sciences v2beta API in addition to Google Cloud Genomics v2alpha1.
More information about migration to the new API from v2alpha1
here.

• Note Google Cloud Life Sciences is the new name for newer versions of Google Cloud Genomics.
• Note Support for Google Cloud Genomics v2alpha1 will be removed in a future version of Cromwell. Advance notice will be provided.

New Docs

Installation methods

Links to the conda package and docker container are now available in
the install documentation.

Bug Fixes

• Fix a bug where zip files with directories could not be imported.
  For example a zip with a.wdl and b.wdl could be imported but one with sub_workflows/a.wdl
  and imports/b.wdl could not.
• Fix a bug which sometimes allowed execution scripts copied by a failed cache-copy to be run instead
  of the attempt-1 script for a live job execution.

48

48 Release Notes

Womtool Graph for WDL 1.0

The womtool graph command now supports WDL 1.0 workflows.

• Note: Generated graphs - including in WDL draft 2 - may look slightly different than they did in version 47.

Documentation

• Documented the use of a HSQLDB file-based database so users can try call-caching without needing a database server.
  Please checkout the database documentation.

47

47 Release Notes

Retry with more memory on Papiv2 (#5180)

Cromwell now allows user defined retries. With memory-retry config you can specify an array of strings which when encountered in the stderr
file by Cromwell, allows the task to be retried with multiplier factor mentioned in the config. More information here.

GCS Parallel Composite Upload Support

Cromwell 47 now supports GCS parallel composite uploads which can greatly improve delocalization performance.
This feature is turned off by default, it can be turned on by either a backend-level configuration setting or
on a per-workflow basis with workflow options. More details here.

Papi V2 Localization Using GCR (#5200)

The Docker image for the Google Cloud SDK was previously only published on Docker
Hub. Now that the image is publicly hosted in
GCR, Papi V2 jobs will localize inputs and delocalize outputs using
the GCR image.

46.1

46.1 Release Notes

Retry with more memory on Papiv2 (#5180)

Cromwell now allows user defined retries. With memory-retry config you can specify an array of strings which when encountered in the stderr
file by Cromwell, allows the task to be retried with multiplier factor mentioned in the config. More information here.

46

46 Release Notes

Nvidia GPU Driver Update

The default driver for Nvidia GPU's on Google Cloud has been updated from 390 to 418.87.00. A user may override this option at anytime by providing the nvidiaDriverVersion runtime attribute. See the Runtime Attribute description for GPUs for detailed information.

Enhanced "error code 10" handling in PAPIv2

On Google Pipelines API v2, a worker VM that is preempted may emit a generic error message like

PAPI error code 10. The assigned worker has failed to complete the operation

instead of a preemption-specific message like

PAPI error code 14. Task was preempted for the 2nd time.

Cromwell 44 introduced special handling that detects both preemption indicators and re-runs the job consistent with the preemptible setting.

Cromwell 46 enhances this handling in response to user reports of possible continued issues.

45.1

[45 hotfix edition] Error code 10 take 2 [BA-5952] (#5138)