Pipeline fails on Jenkins restart

[ipleten]

We have an input stage on lightweight worker which could be sitting very long until user input. So with job waiting for user input and after Jenkins restart if user confirms the input (press "Deploy") we get following error. If I moved out input from module to 'pipeline.groovy' it works, but I expect that some other long running stages might fail after Jenkins restart. It looks like after restart by some reason MPL loses track of activeModules.

We added some debug info so the lines number in stack trace are slightly different, error is happening in https://github.com/griddynamics/mpl/blob/master/vars/MPLModule.groovy#L100

# ./pipeline.groovy
def call(body) {
def MPL = MPLPipelineConfig(body, [modules: []])

pipeline {
    agent none
    stages {
      stage('init') {
        agent none
          steps {
            script { echo "init" }
        }
      }

      stage('user-input') {
        agent none
          steps {
               MPLModule("user-input")
              // If i move content from "user-input" module here everything works.
              //  input message: 'Deploy?', ok: 'Deploy'
          }
      }

      stage('something_else') {
        agent none
        steps {
          script {
            echo 'something_else'
          }
        }
      }
    }
  }
}

Jenkinsfile

@Library(['mpl-library']) _

pipeline {
  modules = [
  "user-input",
  ]
}

module

#user-input.groovy
 input message: 'Deploy?', ok: 'Deploy'

The error I got is :

java.util.NoSuchElementException: Cannot pop() an empty List
at org.codehaus.groovy.runtime.DefaultGroovyMethods.pop(DefaultGroovyMethods.java:8841)
at org.codehaus.groovy.runtime.dgm$483.doMethodInvoke(Unknown Source)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
at org.codehaus.groovy.runtime.callsite.PojoMetaClassSite.call(PojoMetaClassSite.java:47)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
at com.cloudbees.groovy.cps.sandbox.DefaultInvoker.methodCall(DefaultInvoker.java:20)
at com.griddynamics.devops.mpl.MPLManager.popActiveModule(MPLManager.groovy:247)
at MPLModule.call(MPLModule.groovy:104)
at MPLPipelineDeploy.call(MPLPipelineDeploy.groovy:285)
at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.delegateAndExecute(ModelInterpreter.groovy:138)
at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.executeSingleStage(ModelInterpreter.groovy:684)
at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.catchRequiredContextForNode(ModelInterpreter.groovy:418)
at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.catchRequiredContextForNode(ModelInterpreter.groovy:416)
at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.executeSingleStage(ModelInterpreter.groovy:682)
at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.evaluateStage(ModelInterpreter.groovy:281)
at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.toolsBlock(ModelInterpreter.groovy:567)
at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.toolsBlock(ModelInterpreter.groovy:566)
at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.evaluateStage(ModelInterpreter.groovy:271)
at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.withEnvBlock(ModelInterpreter.groovy:466)
at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.withEnvBlock(ModelInterpreter.groovy:465)
at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.evaluateStage(ModelInterpreter.groovy:270)
at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.withCredentialsBlock(ModelInterpreter.groovy:504)
at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.withCredentialsBlock(ModelInterpreter.groovy:503)
at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.evaluateStage(ModelInterpreter.groovy:269)
at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.evaluateStage(ModelInterpreter.groovy:316)
at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.inDeclarativeAgent(ModelInterpreter.groovy:612)
at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.inDeclarativeAgent(ModelInterpreter.groovy:611)
at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.evaluateStage(ModelInterpreter.groovy:313)
at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.stageInput(ModelInterpreter.groovy:379)
at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.stageInput(ModelInterpreter.groovy:378)
at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.evaluateStage(ModelInterpreter.groovy:312)
at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.inWrappers(ModelInterpreter.groovy:635)
at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.inWrappers(ModelInterpreter.groovy:634)
at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.evaluateStage(ModelInterpreter.groovy:252)
at cps.transform(Native Method)
at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:86)
at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:113)
at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixName(FunctionCallBlock.java:78)
at sun.reflect.GeneratedMethodAccessor462.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21)
at com.cloudbees.groovy.cps.Next.step(Next.java:83)
at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:129)
at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:268)
at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18)
at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:51)
at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:185)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:370)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$200(CpsThreadGroup.java:93)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:282)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:270)
at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:67)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Finished: FAILURE

[sparshev]

Hmmm, yeah need to check - but at first looks possible.

[sparshev]

Just returned from vacation - so can work on the issue. Seems like standalone object is not so useful during the pipeline execution - will check how to fix that.

[sparshev]

Sorry, closed by mistake

[sparshev]

So, result of the research (sorry for the delay - really busy) - the standalone objects state is not saved in the program.dat in the active build (which contains the CpsThreadGroup state). For now I see no resolution - because MPLManager is storing the state of the pipeline and right now I see no way to restore the MPLManager instance variable even from the existing MPL variable (which is saved in the program.dat)...

Need to think about it - but for sure it's a blocker for the 1.0.0 version...

[sparshev]

I see 3 potential ways to resolve the issue:

1. Make MPL object before pipeline mandatory (it will be saved in program.dat) and on MPLManager.instance check the CPS nodes on existing object of MPLManager class - and use it to store into a static instance var.
2. Add change to the serializer which can serialize/deserialize - looks hard to do, because it requires to make sure local MPL object and static one are the same or not...
3. Remove the requirement of such an persistent object - potentially possible, but looks hard to do.

[sparshev]

Hi @ipleten ,

Could you please check the provided fix #55

[ipleten]

Pruvit, @sparshev!
We have quick tested changes and jobs are being resumed now.
Thank you!