Jenkins http CLI broken in latest Jenkins LTS releases with anonymous user, can't create ssh users or install plugins

[ChrisPetr0]

🗣️ Foreword

Thank for taking the time to fill this bug report fully. Without it we may not be able to fix the bug, and the issue may be closed without resolution.

👻 Brief Description

Building Jenkins AMI / Image with packer and Ubuntu 20.04. Pattern has been rock solid for years. Truncated Runlist looks like this:

- jenkins_configuration_myorg::install_base_packages
- jenkins_configuration_myorg::install_open_jdk
- jenkins::master
- jenkins_configuration_myorg::jenkins_sudoers
- jenkins_configuration_myorg::install_plugins

No problems until install_plugins recipe. Using the install_plugin resource from Jenkins cookbook which leverages internal Jenkins CLI calls. Using http executor, plugin list is an array walked through for each plugin.

Jenkins LTS 2.319.3 (2022-02-09) works as expected and installs plugins successfully. It seems some security changes were made on later versions of Jenkins that breaks the ability to install plugins with http executor via Jenkins CLI and and anonymous user. Latest Jenkins LTS 2.332.3 and 2.332.2 have this issue (I believe it started in 2.332.1, as significant changes in change log start there).

Error executing action `install` on resource 'jenkins_plugin[ace-editor]'
    amazon-ebs:     ================================================================================
    amazon-ebs:
    amazon-ebs:     Mixlib::ShellOut::ShellCommandFailed
    amazon-ebs:     ------------------------------------
    amazon-ebs:     Expected process to exit with [0], but received '6'
    amazon-ebs:     ---- Begin output of "java" -jar "/tmp/packer-chef-solo/local-mode-cache/cache/jenkins-cli.jar" -s http://localhost:8080 -http install-plugin https://updates.jenkins.io/download/plugins/ace-editor/1.1/ace-editor.hpi  ----
    amazon-ebs:     STDOUT:
    amazon-ebs:     STDERR: ERROR: anonymous is missing the Overall/Read permission

I'm perfectly fine with using ssh executor and creating a user with public key and setting node.run_state[:jenkins_private_key] = private_key. In fact, I tried this. However, the same CLI problem exists. If I have a base install of Jenkins via this cookbook, want to configure a user via this cookbook via jenkins_user resource, that also needs to use the CLI. I can't create a Jenkins user with ssh configured via this cookbook / resource to use the CLI with ssh executor because it has the same problem when calling the jenkins_user resource (ERROR: anonymous is missing the Overall/Read permission.

In testing, I also tried to modify

<denyAnonymousReadAccess>false</denyAnonymousReadAccess> 

and run CLI calls with http executor via anonymous user and the error changes to ERROR: anonymous is missing the Overall/Administer permission

I'm thinking I may have to resort to trying to run CLI calls with -auth flag and use the admin user with $JENKINS_HOME/secrets/initialAdminPassword which I did try unsuccessfully but need to test more. EDIT: This works, going to try to figure out if I can use this pattern for initial configuration.

I would really like to get on latest LTS version, but it seems there is a chicken and egg problem here with CLI unable to be used at all until an ssh user is created, but this cookbook can't create an ssh user because the CLI no longer works with anonymous user.

🥞 Cookbook version

'9.5.2'

👩‍🍳 Chef-Infra Version

17.9.26

🎩 Platform details

Ubuntu 20.04

Steps To Reproduce

Steps to reproduce the behavior:

• Set Jenkins Version to 2.332.3
• Instantiate Jenkins node configuration from scratch with default cookbook settings (after installing java) using Jenkins[master] recipe (http executor)
• Leverage install_plugin resource and attempt to install a valid plugin
• Notice failure with STDERR: ERROR: anonymous is missing the Overall/Read permission
• Can also reproduce with jenkins_user, groovy_script, and jenkins_command, basically anything that uses internal CLI to configure Jenkins using http executor with no auth

EDIT: Upon reading cookbook documentation further, I am going to try the jenkins_credentials but unfamiliar with using that. I'm not sure if this will help get around this issue, but will post back.

EDIT_UPDATE: Looks like:

jenkins_password_credentials 'cliuser' do
  id          'cliuser'
  description 'cliuser'
  password    'clipass123'
end

also fails and uses the CLI to run a groovy script

amazon-ebs:     ================================================================================
    amazon-ebs:     Error executing action `create` on resource 'jenkins_password_credentials[cliuser]'
    amazon-ebs:     ================================================================================
    amazon-ebs:
    amazon-ebs:     Mixlib::ShellOut::ShellCommandFailed
    amazon-ebs:     ------------------------------------
    amazon-ebs:     Expected process to exit with [0], but received '6'
    amazon-ebs:     ---- Begin output of "java" -jar "/tmp/packer-chef-solo/local-mode-cache/cache/jenkins-cli.jar" -s http://localhost:8080 -http groovy = ----
    amazon-ebs:     STDOUT:
    amazon-ebs:     STDERR: ERROR: anonymous is missing the Overall/Read permission

• Set Jenkins Version to 2.319.3
• Perform exact same test as above and watch it install plugins with no issues

🚓 Expected behavior

Jenkins CLI calls leveraged in install_plugin or jenkins_user resources should work or a method to create Jenkins ssh user using an alternative method than Jenkins CLI should exist.

➕ Additional context

I'm not really sure what to do here. It seems like I am deadlocked from using the functionality of this cookbook natively without introducing additional yet to be determined code to create an ssh enabled user or use jenkins admin user auth flag with http executor to create ssh enabled users, which will require custom code.

Any help, suggestions, or workarounds would be greatly appreciated. I'm looking to keep up with LTS releases from a security perspective. If anything else is needed from me, I will be happy to provide.

Thank you !

[ChrisPetr0]

Here's a shell script I used with packer (directly before install_plugins) to get around this issue. Note local Jenkins users are not used in final deployment.

#!/bin/bash -x

cd /var/lib/jenkins

cli_groovy=cli_user.groovy

sudo cp /tmp/${cli_groovy} /var/lib/jenkins/${cli_groovy}

echo | systemctl | grep jenkins

sudo wget http://localhost:8080/jnlpJars/jenkins-cli.jar

pw=`sudo cat secrets/initialAdminPassword`

plugins=("matrix-auth")

for str in ${plugins[@]}; do
  sudo java -jar jenkins-cli.jar -auth admin:$pw -s http://localhost:8080 install-plugin $str
done

sudo systemctl stop jenkins
sleep 10
sudo systemctl start jenkins
sleep 30

sudo chmod 777 /var/lib/jenkins/${cli_groovy}
sudo chown jenkins:jenkins /var/lib/jenkins/${cli_groovy}

sudo java -jar jenkins-cli.jar -auth admin:$pw -s http://localhost:8080 groovy = < /var/lib/jenkins/${cli_groovy}

sudo systemctl stop jenkins
sleep 10
sudo systemctl start jenkins
sleep 30

[djessich]

I got the exact same problem:

ERROR: anonymous is missing the Overall/Read permission

See:

I can confirm, on LTS 2.319.3 everything works fine. Latest version does not work anymore.

[AdrianVarelaG]

I have the same problem using centos-7. i can authenticate using "admin" and the content of secrets/initialAdminPassword.
I try to set:
node.run_state[:jenkins_username] = 'vagrant'
node.run_state[:jenkins_password] = 'vagrant'
but didn't work.

[djessich]

It is very annoying that this is still unresolved. There is no specific guide on how to solve this and results in broken Chef runs. So any ideas how this can be solved?

[ChrisPetr0]

It is very annoying that this is still unresolved. There is no specific guide on how to solve this and results in broken Chef runs. So any ideas how this can be solved?

My problem was installing plugins. I posted how to solve it in my initial response. What are you trying to do? What tools are you using to run chef? Are you creating a new image or trying to converge an existing host?

[skies5]

Same problem here.

It looks like Jenkins is ignoring the "runSetupWizard=false" flag. When opening the Jenkins instance in any browser, the Jenkins unlock screen is shown, which should NOT be the case. (This explains why unauthenticated CLI-calls fail, because a locked Jenkins does not accept these yet.)
I tried setting the flag manually, but that did not work either.

So my conclusion: either the runSetupWizard-flag is not delivered to Jenkins at all, or the flag itself is bugged in Jenkins 2.332+.

[ChrisPetr0]

Same problem here.

It looks like Jenkins is ignoring the "runSetupWizard=false" flag. When opening the Jenkins instance in any browser, the Jenkins unlock screen is shown, which should NOT be the case. (This explains why unauthenticated CLI-calls fail, because a locked Jenkins does not accept these yet.) I tried setting the flag manually, but that did not work either.

So my conclusion: either the runSetupWizard-flag is not delivered to Jenkins at all, or the flag itself is bugged in Jenkins 2.332+.

None of the implementation on main of this repo basically works at this point. They changed Jenkins to use a different service manager in linux noted here: #783 , that's why it doesn't see the flag unless you use the workaround in that bug

[djessich]

Jenkins switched from sysvinit to systemd in recent versions. As this cookbook still configures the sysvinit/initd way, the Jenkins options and Jenkins environment configuration is not passed to Jenkins at all. This causes Jenkins to start with the default options provided by the RPM package. I can confirm that the implementation of this cookbook does not work with latest Jenkins versions anymore.

The workaround I use currently is as follows:
I do not install Jenkins using the master recipe of this cookbook anymore, as I cannot add code to the recipe before the service is started to configure the systemd unit. This is a limitation of Chef, as it is not possible to extend a recipe in Chef. So I copied the recipe to my wrapper cookbook and applied my changes to it. I now install the Jenkins packages (including font packages), configure the systemd unit (under /etc/systemd/system/jenkins.service as this overrides the one provided by RPM package), remove all sysvinit configs (still delivered with Jenkins RPM and configured by this cookbook) and start / enable the Jenkins service. And voila, everything works as expected now, but using systemd instead of sysvinit/initd. Also all resources of this cookbook work like a charm 🚀

[Stromweld]

We'd be happy to review PR's if you'd like to propose those changes. We can then release as a new major version of the cookbook so those that aren't using latest Jenkins with systemd can pin to the current version.