[RFD - 0004] - Semaphore Packages

[AleksandarCole]

[RFD - 0004] - Semaphore Packages

Authors	State
@darkofabijan @radwo @DamjanBecirovic @lucaspin	Draft

Table of Contents

• What
• Why
• Solutions
  • Overview
  • Using Semaphore Packages
    • Using public packages from the official registry
    • Using third-party open-source packages
    • Using organization level private packages
    • Using local packages from your repository
  • Customizing imported CI/CD elements
    • Passing arguments to imported steps, blocks, and pipelines
    • Customizing by overriding configuration
    • Passing arguments to the whole package
  • Creating a Semaphore Package
    • Creating your first package
    • Exporting CI/CD components
  • Exporting steps
    • Exporting Blocks
    • Exporting Pipelines
    • Types of Arguments
  • Publishing packages
    • Publishing an open-source package to the Package Registry
    • Adding a private package to your Semaphore organization
  • New CI/CD concepts
    • Steps
    • Sub-Pipelines
    • YAML Compilation
• Package Registry/Inventory
• UI integration
• Security

What

Semaphore Packages are a framework designed to create and share common CI/CD elements between your pipelines. They enable developers to encapsulate repeatable steps, configurations, and scripts into reusable packages. These packages can be versioned, maintained, and distributed across multiple projects.

Why

Key concerns solved by Semaphore Packages:

• Codifying CI/CD practices in organizations: As the number of projects grows in an organization, the need for uniformity increases. Semaphore Packages allow you to codify common steps in your pipelines. Changes in a package reflect across all projects.
• Fast CI/CD pipeline booststrapping: To preserve productivity and developer happiness, the path from an idea to production should be as simple as possible. Bootstrapping a CI/CD pipeline is crucial. Semaphore Packages allow you to leverage the open-source community and use off-the-shelf solutions.
• CI/CD pipeline stability: There is a gap between a working pipeline and a stable & fast pipeline. By using packages, you ensure that Caching, Network Failure handling, Retries, and extensive logging are built into your pipeline.

---

Functionality

Overview

Packages are modeled after code libraries, where a set of pre-defined functionality can be included in an application. Similarly, Semaphore Packages are a set of pre-defined CI/CD components.

A Semaphore package is a collection of steps, blocks, and pipelines that can be imported into a pipeline.

An example Semaphore Package that exports a complex Android Emulator step is defined with:

version: 1
name: Android
description: Steps for starting and managing Android emulators.
export:
  steps:
    - name: ChangeEmulatorVersion
      args:
        - name: PlatformVersion
          default: android-29
          type: string
        - name: BuildToolVersion
          default: 30.0.0
          type: string
      spec: >
        platform="platforms;{{ PlatformVersion }}"

        tools="build-tools;{{ BuildToolVersion }}"

        system="system-images;{{ PlatformVersion }};google_apis;x86"

        emulator_opts="-noaudio-no-boot-anim -gpu off -no-window"


        sdkmanager "platform-tools" "$platform" "$tools" "emulator"

        sdkmanager "$system"

         echo no | avdmanager create avd -n test-emulator -k "$system"
        emulator -avd test-emulator $emulator_opts &


        wait_cmd='while [[ -z $(getprop sys.boot_completed) ]]; do sleep 1;
        done;'

        adb wait-for-device shell $wait_cmd

The above YAML configuration defines steps for changing the Android emulator version.

This package can be included in your YAML definition with:

version: 1
name: CI
include:
  - from: semaphoreci/android
blocks:
  - name: Tests
    jobs:
      - name: Unit Test
        steps:
          - use: Android.ChangeEmulatorVersion
            args:
              PlatformVersion: 29.0.0
              BuildToolVersion: 30.0.0

Using Semaphore Packages

Using public packages from the official registry

The official package registry contains a set of officially supported packages that you can use in your pipelines.

Packages include steps for working with major frameworks (ex. Rails, Express, React), language-specific steps (ex. Go, Rust), and common steps for deployment (ex. GCP, AWS, Heroku).

The official packages are name spaced under the semaphoreci/* namespace.

An example pipeline that uses the official Rails package to include blocks for running Rubocop, Brakeman, RSpec, and Cucumber would be specified with this YAML:

version: 1.0
include:
  - from: semaphoreci/rails

blocks:
  - use: Rails.Lint
  - use: Rails.SecurityScan
  - use: Rails.RSpec
     args:
        parallelism: 4
   - use: Rails.Cucumber
     args:
        parallelism: 4

Using third-party open-source packages

The official registry also hosts third party packages, contributed by third-party developers.

Third-party packages are name spaced under the name of the contributor <name>/*.

An example pipeline that uses third-party packages developed by jimbo that exports CI metrics to DataDog are imported like in the following example:

version: 1
include:
  - from: jimbo/datadog
blocks:
  - name: Tests
    task:
      jobs:
        - name: Unit Tests
          steps:
            - make test
      epilogue:
        always:
          steps:
            - use: Datadog.ExportCIMetrics
              args:
                datadogSecretName: datadog-access-tokens

Using organization level private packages

Every organization has a private package namespace accessible only by pipelines in that organization. Private Packages are ideal for modeling internal conventions and reusing common CI/CD elements between your projects.

Packages in the organization are not namespaced.

An example package that exports a common CI/CD steps for Lint, Unit, and Security Scans for microservices can be used with:

version: 1
include:
  - from: microservices
block:
  - use: Microservices.Lint
  - use: Microservices.SecurityScan

Using local packages from your repository

Packages can be included directly from the repository. Local Packages are ideal for abstracting parts of the pipeline, keeping the main pipeline shorter and more concise.

An example package that export steps for deployment can be used with:

version: 1
include:
  - from: ./deployment/k8s.yml
blocks:
  - use: K8s.Apply
    args:
      path: k8s/**/*.yml
  - use: K8s.WaitForRollout

The package in the above example is imported from a location relative to the Pipeline YAML. If the Pipeline YAML is in .semaphore/prod.yml, the include path ./deployment/k8s.yml will load it from .semaphore/deployment/k8s.yml.

To access a package relative to the root of the repository, start the from stanza with /:

version: 1
include:
  - from: /ci_packages/k8s.yml
blocks:
  - use: K8s.Apply
    args:
      path: k8s/**/*.yml
  - use: K8s.WaitForRollout

The package is loaded from <repo-root>/ci_package/k8s.yml.

Customizing imported CI/CD elements

The included packages can be customized to fit the need of the current pipeline. Packages allow three types of customization:

1. Customizing by receiving arguments
2. Customizing by overriding configuration
3. Passing arguments to the whole package

Passing arguments to imported steps, blocks, and pipelines

Every package can define a set of arguments.

Arguments enable package creators to receive configuration options. For package users, this is the primary way of customizing the CI/CD element.

In the following example, we will use the Golang package and customize the Test block to run tests with verbose output:

version: 1
include:
  - from: semaphoreci/golang
blocks:
  - use: Golang.Test
    args:
      verbose: true

Customizing by overriding configuration

Apart from passing arguments, the package users can also opt for overriding stanzas in the included blocks.

In the following example, we will override the Heroku deployment block's prologue steps to execute several steps before running the deployment and inject deployment secrets.

version: 1
include:
  - from: semaphoreci/heroku
blocks:
  - use: Heroku.Deploy
    secrets:
      - heroku-deployment-secrets
    prologue:
      steps:
        - make compile.assets

Passing arguments to the whole package

Package level arguments allow you to pass values to every exported CI/CD element.

In the following example, we will use the goRoot argument that will be applied on every exported element from that package:

version: 1
include:
  - from: semaphoreci/golang
    args:
      goRoot: /tmp/go-root/
block:
  - use: Golang.Lint
  - use: Golang.FmtCheck
  - use: Golang.Tests

Creating a Semaphore Package

Creating your first package

Semaphore Packages are a collection of steps, blocks, and pipelines that can be used in your CI/CD pipelines.

In this example, we are constructing a barebone Hello World package. When included in a pipeline, the block will print a message in the output.

Create a .semaphore/packages/hello.yml file with the following content:

version: 1
name: HelloWorld
description: Utilities for displaying messages in the terminal.
export:
  steps:
    - name: PrintMessage
      args:
        - name: message
          type: string
          default: Hello World
          description: Message that will be displayed in the output.
      spec: |
        echo {{ message }}

We have created our first package!

• It exports one step called PrintMessage
• The PrintMessage step accepts one argument - The message that we want to display
• The default value of the message is "Hello World"

To use it in a pipeline, include it in your Semaphore YAML configuration:

version: 1.0
include:
  - from: ./packages/hello.yml # <-- Included the package

blocks:
  - name: Example
    jobs:
      steps:
        - use: HelloWorld.PrintMessage # <-- Using the PrintMessage step
          args:
            message: "Hello Packages!" # <-- Providing arguments to the step

Exporting CI/CD components

The Hello World package demonstrated how to export a step from a package. Steps are the simplest construction elements, but they are not the only onse.

Semaphore Packages allow you to export three types of elements. Steps, Blocks, and Pipelines.

• Steps are the simplest. They model concrete steps in the context of a job. Example are:
  StartDatabase, StartWebserver, ChangeFirefoxVersion, RunTest.
• Blocks are a bigger piece of a CI/CD pipeline. Think of "run 3 parallel jobs with unit
  tests", "execute static code analysis", "release a docker image".
• Pipelines are a collection of blocks. They are ideal for modeling complex flows. Think
  of "blue-green deployment", "CI steps".

To explore the whole suite of options, we will write a Heroku package from scratch.

Exporting steps

Start by creating a ./semaphore/packages/heroku.yml in your repository.

version: 1.0
name: Heroku
description: Manage and release your Heroku applications.
args:
  - name: HerokuSecret
    type: secret
    description: The secret that contains the Heroku Access Token.

  - name: AppName
    type: string
    description: Name of your Heroku application.

  - name: AppHealthCheckEndpoint
    type: string
    description: The path that checks if your application is online.

export:
  steps: #TODO
  blocks: # TODO
  pipelines: # TODO

We have set up the base of our new package for managing Heroku applications.

Time to export some Heroku specific steps from the ./semaphore/packages/heroku.yml package.

export:
  steps:
  - name: InstallHerokuCLI
    spec: "curl https://cli-assets.heroku.com/install.sh | sh"

  - name: Configure
    spec: |
      if [[ -z $HEROKU_API_KEY ]]; then
        echo "(error) HEROKU_API_KEY is not configured."
        echo ""
        echo "Please add a secret with HEROKU_API_KEY environment variable"
      else
        echo "Adding Heroku Remote"
        remote="https://heroku:$HEROKU_API_KEY@git.heroku.com/{{ AppName}}.git"
        git remote add $remote
      fi

  - name: Deploy
    args:
    - name: force
      type: boolean
      description: If set, force push will be used to deploy to Heroku.
      default: false
    spec: |
      if {{ force }}; then
        force="-f"
      fi
      git push $force heroku $SEMAPHORE_BRANCH_NAME:master

  - name: MigrateDB
    spec: heroku run --app {{ AppName }} rake db:migrate

  - name: RestartApp
    spec: heroku ps:restart

  - name: HealthCheck
    spec: retry -t 10 'curl --fail {{ AppHealthCheckEndpoint }}'

To use these steps, you can include them into the pipeline:

include:
  - from: ./packages/heroku.yml
    args:
      HerokuSecret: heroku-prod
      AppName: zipzap
      AppHealthCheckEndpoint: https://zipzap.com/health

prologue:
  steps:
    - checkout
    - use: Heroku.InstallHerokuCLI
    - use: Heroku.Configure
blocks:
  - name: Deploy
    jobs:
  - name: Deploy
    steps:
      - use: Heroku.Release
      - use: Heroku.Migrate
      - use: Heroku.HealthCheck

Exporting Blocks

Steps offer the most flexibility, but they require a bit of know-how to use in your pipelines. To bundle and export our steps into a bigger logical chunk, we can use blocks.

Going back to our .semaphore/package/heroku.yml and combining our steps into blocks:

export:
  blocks:
    - name: Release
      jobs:
        - name: Release
          steps:
            - use: this.InstallHerokuCLI
            - use: this.Configure
            - use: this.Deploy

    - name: Migrate
      jobs:
        - name: Migrate DB
          steps:
            - use: this.InstallHerokuCLI
            - use: this.Configure
            - use: this.Migrate

    - name: Verify
      jobs:
        - name: HealthCheck
          steps:
            - use: this.HealthCheck

To use them in a Pipeline:

include:
  - from: ./packages/heroku.yml
    args:
      HerokuSecret: heroku-prod
      AppName: zipzap
      AppHealthCheckEndpoint: https://zipzap.com/health
      
blocks:
  - use: Release
  - use: Migrate
  - use: Verify

Exporting Pipelines

To go even a level beyond, we can combine and export a full pipeline to our users.

This option is the fastest to use but offers the least amount of flexibility.

export:
  pipelines:
   - name: DeployToHeroku
     prologue:
       steps:
         - checkout

     blocks:
       - name: "Verify Health Before Release"
         use: this.Verify

       - use: this.Relase
       - use: this.Migrate

       - name: "Verify Health After Release"
         use: this.Verify

To use it in our pipelines:

include:
  - from: ./packages/heroku.yml
    args:
      HerokuSecret: heroku-prod
      AppName: zipzap
      AppHealthCheckEndpoint: https://zipzap.com/health

use: DeployToHeroku

Types of Arguments

The Semaphore Packages framework accepts the following argument types:

• Integers
• Strings
• Booleans
• SecretName
• EnvironmentVariableName
• WhenCondition
• StepList
• Agent

Some argument types can provide optional validation parameters:

Integers
Integers can set a valid range of values. For example:

name: Parallelism
type: integer
range: 1..30
default: 5

String

Strings can set a minimal and maximal length. For example:

name: Username
type: string
length: 0..100

Strings can also set a regex validation pattern:

name: Username
type: string
regex: "/^[A-Za-z0-9]+$/"

Strings can also define a list of valid values:

name: ReadyState
type: string
oneOf:
  - "deployed"
  - "ready"
  - "running"
default: "deployed"

Publishing packages

A central package registry allows fast access to a broad set of Semaphore Packages. Packages are developed in a Git repository and published to the registry.

Publishing an open-source package to the Package Registry

To publish a package to the registry, execute the following command:

$ sem create package --public --source packages/hello.yml

To verify that the package was uploaded, execute:

$ sem get packages --public --mine
NAME LATEST VERSION EXPORTS
shiroyasha/hello v1.1 1 step, 1 block

To verify that the package details are correct, execute:

$ sem describe package hello

PACKAGE

Name: HelloWorld
Version: v1.1

Display Hello World with a custom message.

USAGE

1) Install the package
 
  include:
    - name: HelloWorld
      from: shiroyasha/hello

2) Use the hello world block to display a message

blocks:
    - name: Hello
    use: HelloWorld.Display
    params:
      message: Hello Hello Hello

EXPORTED BLOCKS

1) Display - Displays a message on the command line

    Params:
      - message - The message to display, default: Hello World, type: string

Adding a private package to your Semaphore organization

Publishing a private — organization scoped package — is analogues to publishing a public package.

To publish a package to the registry, execute the following command:

$ sem create package --private --source packages/hello.yml

To verify that the package was uploaded, execute:

$ sem get packages --private
NAME LATEST VERSION EXPORTS
hello v1.1 1 step, 1 block

To verify that the package details are correct, execute:

$ sem describe package hello

PACKAGE

Name: HelloWorld
Version: v1.1

Display Hello World with a custom message.

USAGE

1) Install the package
 
  include:
    - name: HelloWorld
      from: hello

2) Use the hello world block to display a message

blocks:
    - name: Hello
    use: HelloWorld.Display
    params:
      message: Hello Hello Hello

EXPORTED BLOCKS

1) Display - Displays a message on the command line

    Params:
      - message - The message to display, default: Hello World, type: string

New CI/CD concepts

The introduction of Semaphore Packages comes bundled with a handful of new CI/CD
concepts.

Steps

Steps are the new atomic instructions in a CI/CD pipeline. They are a generalization of commands and aim to resolve the shortcomings of the current system.

A step can be a simple bash command:

steps:
  - export FOO=bar
  - echo $FOO

It can also be a multiline command with a name:

steps:
  - name: Prepare Database
    commands: |
      if [ $DB_TYPE = 'postgres']; do
        sem-service start postgres
      else
        sem-service start mysql
      fi
      bundle exec rails db:create

Bash is not the best tool for every CI/CD step. For example, managing maps is notoriously hard in Bash. Steps can use a custom scripting language, for example, Ruby:

steps:
  - name: Notify the team on Slack
    executeWith: ruby
    commands: |
      require 'json'

      message = {
        "title" => "Hello",
        "message" => "Hello from a Semaphore Step"
      }.to_json

      Net::HTTP.start(uri.hostname, uri.port) do |http|
        req = Net::HTTP::Post.new(URI(ENV["SLACK_ENDPOINT"]))
        req.body = message.to_json
        http.request(req)
      end

Sub-Pipelines

Sub-pipelines reduce the overhead of managing complex projects with many interconnected blocks. Nowhere is this more apparent than in Monorepo projects with multiple services triggered with a change_in filter.

A use case is demonstrated in the next example that uses sub-pipelines to define steps for each service in a monorepo context:

version: 1.0
blocks:
  - name: Service A
    when: change_in("service1/**/*")
    pipeline:
      blocks:
        - name: Lint
          jobs: ... # shortened for brevity
        - name: Unit Tests
          jobs: ... # shortened for brevity

  - name: Service B
    when: change_in("service2/**/*")
    pipeline:
      blocks:
       - name: Lint
          jobs: ...
       - name: Unit Tests
          jobs: ...

  - name: Release
    dependancies: ["Service A", "Service B"]
    jobs:
      - name: Release
        steps:
          - make release

YAML Compilation

Semaphore aims to remain transparent and easy to debug. With Semaphore Packages, the YAML fetching and compilation steps are exposed in the init job.

In case of problems, the developer can peak into the compilation logs and trace out if a dependency is causing problems.

---

Package Registry/Inventory

🔧 work in progress 🔧

Functional requirements for public and private package registries as well as details on versioning, packages documentation and package structure are to be described here.

UI integration

🔧 work in progress 🔧

UI/UX updates for integration with the Workflow Builder and UI design of the Packages Registry/Inventory to be described here.

Security

🔧 work in progress 🔧

Security requirements, architecture and threat assessment to be described here.