Jenkins for Pipelines

Intro

pipelines playbook includes a role to install configure and use jenkins for pipelines data processing jobs.

Additional steps

Extra plugins requirements

Follow the role README to install some jenkins plugins dependencies.

Add ssh credentials

(...)

Add the Jenkins slaves

(...)

Script approvals

Some of these preconfigured jobs will need some extra script approval during their first Run:

Navigate to jenkins > Manage jenkins > In-process Script Approval

Verify master label

Navigate to jenkins > Manage jenkins > Configure > Labels and verify that your jenkins has a correct Label that match your spark and pipelines node name