0.21

Cromwell Change Log

0.21

• Warning: Significant database updates when you switch from version 0.19 to 0.21 of Cromwell.
  There may be a long wait period for the migration to finish for large databases.
  Please refer to MIGRATION.md for more details.
• There are significant architectural changes related to increases in performance and scaling.
• The biggest user-facing changes from 0.19 to 0.21 are related to the application.conf file, which has been restructured significantly.
  The configuration for backends now is all contained within a backend stanza, which specifies 1 stanza per name per backend and a default backend, as follows:

backend {
    default=Local
    providers {
        Local {
            actor-factory = "cromwell.backend.impl.sfs.config.ConfigBackendLifecycleActorFactory"
            config {
                ... backend specific config ...
            }
        }
        JES {
            actor-factory = "cromwell.backend.impl.jes.JesBackendLifecycleActorFactory"
            config {
                ... backend specific config ...
            }
        }
        SGE {
            actor-factory = "cromwell.backend.impl.sfs.config.ConfigBackendLifecycleActorFactory"
            config {
                ... backend specific config ...
            }r
        }
    }
}

• A new /stats endpoint has been added to get workflow and job count for a Cromwell running in server mode.
• Renamed Workflow Options:
  “workflow_log_dir” -> “final_workflow_log_dir”
  “call_logs_dir” -> “final_call_logs_dir”
  “outputs_path” -> “final_workflow_outputs_dir”
  “defaultRuntimeOptions” -> “default_runtime_attributes”
• Timing diagrams endpoint has been updated to include additional state information about jobs.
• Add support for Google Private IPs through noAddress runtime attribute. If set to true, the VM will NOT be provided with a public IP address.
  Important: Your project must be whitelisted in "Google Access for Private IPs Early Access Program". If it's not whitelisted and you set this attribute to true, the task will hang.
  Defaults to false.
  e.g:

task {
    command {
        echo "I'm private !"
    }

    runtime {
        docker: "ubuntu:latest"
        noAddress: true
    }
}

• The Local and the SGE backend have been merged into a generic
  Shared File System (SFS) backend. This updated backend can be configured
  to work with various other command line dispatchers such as LSF. See the
  README for more info.
• On the JES and SFS backends, task command blocks are now always
  passed absolute paths for input Files.
• On the SFS backends, the call directory now contains two sub-directories:
  • inputs contains all the input files that have been localized for this task (see next below for more details)
  • execution contains all other files (script, logs, rc, potential outputs etc...)
• Override the default database configuration by setting the keys
  database.driver, database.db.driver, database.db.url, etc.
• Override the default database configuration by setting the keys
  database.driver, database.db.driver, database.db.url, etc.

For example:

# use a mysql database
database {
  driver = "slick.driver.MySQLDriver$"
  db {
    driver = "com.mysql.jdbc.Driver"
    url = "jdbc:mysql://host/cromwell"
    user = "user"
    password = "pass"
    connectionTimeout = 5000
  }
}

0.20

• The default per-upload bytes size for GCS is now the minumum 256K
  instead of 64M. There is also an undocumented config key
  google.upload-buffer-bytes that allows adjusting this internal value.
• Updated Docker Hub hash retriever to parse json with custom media
  types.
• Added a /batch submit endpoint that accepts a single wdl with
  multiple input files.
• The /query endpoint now supports querying by id, and submitting
  parameters as a HTTP POST.

0.19.3

• Fix an issue with symbolic linking localization strategy. Intermediate directories were not being created.
• Add LSF backend

0.19

• Workflow options may now contain a choice of backend specific for that workflow:

{
  "backend": "JES"
}

• To support the above change, the configuration file has changed how backends are specified. You must replace backend.backend with backend.defaultBackend.
  In addition the option backend.backendsAllowed must be specified (and should include the default), for example:

backend {
  defaultBackend = "local"
  backendsAllowed = [ "local", "JES", "SGE" ]
  ...

• New runtime option for JES: bootDiskSizeGb. Allows specification of a boot disk size (as an Integer number of GB) that can be increased to boot a larger docker image.
• Workflow options now allows you to specify a workflowFailureMode to control workflow behavior after a call has failed, for example:{ "workflowFailureMode": "..." }. The options are:
  • ContinueWhilePossible - continues to start and process calls in the workflow, as long as they did not depend on the failing call
  • NoNewCalls - no new calls are started but existing calls are allowed to finish
  • The default is NoNewCalls but this can be changed using the workflow-options.workflow-failure-mode configuration option.
• Bug fix: Tasks that changed directory would fail on JES because their return code file was written to the new directory instead of an absolute path
• Bug fix: Using write_* functions in a Task's command (e.g. ./my_script --file=${write_file(my_array)}) will now work with JES
• Changing format of the 'disks' runtime attribute slightly to allow for mounting disks at specific mountpoints

task disk_test {
  command { ... }
  runtime {
    disks: "local-disk 20 SSD, /mnt/mnt1 200 HDD"
  }
}

• Metadata now contains a list of failures for calls and workflows. This is an optional element of both call and workflow and is shaped thus:

"failures": [
  {
    "failure": "The failure message",
    "timestamp": "2016-02-25T10:49:02.066-05:00"
  }
]

• Added workflow options to copy the call logs and/or the workflow logs to a call_logs_dir or a workflow_log_dir,
  respectively.
• The system properties LOG_MODE and LOG_LEVEL used by Logback may now be specified as environment variables.
• Implemented write_tsv function
• Support sub function from the WDL Standard Library. See https://github.com/broadinstitute/wdl/blob/subFunction/SPEC.md#string-substring-string-string

0.18

• The deprecated parse, validate, inputs and highlight functionality from the command line tool has been removed in favor of wdltool (https://github.com/broadinstitute/wdltool)
• Workflow options can now include a defaultRuntimeOptions section, so that the same runtime attribute is not needed in every single WDL task. E.g.:

{
  "defaultRuntimeOptions": {
    "docker": "ubuntu:latest"
  }
}

• Changed the JES runtime attributes defaultDisks and defaultZones to be simply disks and zones respectively.
• Liquibase scripts now run automatically. Non-persistent, in-memory databases are not affected. However Cromwell will
  not start if it detects evidence of manually run liquibase migrations in a persistent database. Instead, before Cromwell
  will start cleanly, the database should backed up, and then this SQL should be manually executed:

update DATABASECHANGELOG
  set MD5SUM = null,
    FILENAME = substr(FILENAME, instr(FILENAME, "src/main/migrations/") + length("src/main/migrations/"))
  where FILENAME like '%src/main/migrations/%'

• Added Preemptible VMs support for JES. This has impacts on the API Endpoint responses as a Call/Shard can now be attempted multiple times. Each attempt will have its own entry.
• Added custom thread pool to workaround Slick deadlocks. The thread pool
  size defaults to the Slick configuration value db.numThreads, but may be increased up to Slick's
  db.maxConnections, via a new property actionThreadPoolSize.
• Added support for size WDL standard library function.
• Allow for runtime attribute values to be interpreted as full expressions. For example:

task example {
  String ubuntu_tag
  command { ... }
  runtime {
    docker: "ubuntu:" + ubuntu_tag
  }
}

• Add runtime attributes in Call metadata :

{
  "workflowName": "hello",
  "calls": {
    "hello.hello": [
      {
        ...,
        "runtimeAttributes": {
                  "preemptible": "0",
                  "failOnStderr": "false",
                  "disks": "local-disk 10 SSD",
                  "continueOnReturnCode": "0",
                  "docker": "ubuntu:latest",
                  "cpu": "1",
                  "zones": "us-central1-a",
                  "memory": "2GB"
                },
        ...
      }
    ]
  }
}

• Added "preemptible" field in Call metadata. This only appears if the backend is JES.

{
  "workflowName": "hello",
  "calls": {
    "hello.hello": [
      {
        ...,
        "preemptible": "true"
        ...
      }
    ]
  }
}

0.17

• Validate RuntimeAttributes before starting a call
• Prevent infinite retries
• Persist disallowed return codes in DB
• Add deprecation notices to WDL specific command line functionality
• Make Call backoff polling backend specific
• Aborts jobs on shutdown.
• Numerous bug fixes and stability improvements

0.16

• Bug fix: call caching will not calculate MD5s if turned off
• New API endpoint for generating HTML timing diagrams of a workflow
• Move the Scala bindings for WDL into a new project wdl4s

0.15

• Call caching allows Cromwell to use results from prior invocations if it detected that it's run a call in the past
• Various bug fixes, stability improvements
• Improved REST API

0.14

• Improved logging
• Improved concurrency handing for high workflow loads
• Bug fixes
• Improved stability

0.13

• Stability and bug fixes

• implemented read_float(), read_boolean(), read_object(), and write_object() WDL functions

• Allow WDL functions at the workflow level:

  workflow wf {
    Array[Int] my_ints = read_lines("some_file")
  }
  

0.12

Merge branch 'develop'