Azure pipeline implementation for portman framework

[shubhambajad]

Hi Team,

We have use Portman for our API automation framework and we have automate all our APIs and now we want to implement azure pipeline for a execution purpose so any inputs for the same like how we can implement that or what we can use in it so that all execution should get happen appropriately?

[thim81]

Hi @shubhambajad

Portman is a NodeJS CLI package, so there are multiple ways to run them in an Azure Pipeline:

• via a command/script in your package.json file
• via NPX

Example via package.json

package.json >>

  "scripts": {
    "portman": "portman --cliOptionsFile configs/openapi-to-newman.json",
}

and next you setup a Azure Pipeline YAML file which contains the typical steps:

stages:
  - stage: run_portman_tests
    displayName: Run Portman tests
    jobs:
      - job:
        displayName: 'Execute Portman'
        steps:
          - checkout: self
            displayName: Git checkout
            persistCredentials: false
            submodules: true

          - task: NodeTool@0
            displayName: Use Node 16.x
            inputs:
              versionSpec: 16.x

          - bash: npm ci
            workingDirectory: './api'
            displayName: NPM ci

          - bash: npm run portman
            workingDirectory: './api'
            displayName: 'Run portman'

Example via NPX:

stages:
  - stage: run_portman_tests
    displayName: Run Portman tests
    jobs:
      - job:
        displayName: 'Execute Portman'
        steps:
          - checkout: self
            displayName: Git checkout
            persistCredentials: false
            submodules: true

          - task: NodeTool@0
            displayName: Use Node 16.x
            inputs:
              versionSpec: 16.x

          - bash: npm ci
            workingDirectory: './api'
            displayName: NPM ci

          - bash: npx -y portman --cliOptionsFile configs/openapi-to-newman.json
            workingDirectory: './api'
            displayName: 'Run portman'

The biggest difference is that NPX will download the latest version of Portman before executing the Portman CLI, while with the package.json example, this will respect the dependency versioning for Portman.

Advanced example

The example below is more advanced with Azure reporting & storing the result as an artifact.
The teams that I work with use Azure Pipeline to run API tests via Portman & Newman. We run them convertion & newman test run as seperate steps.

Below is an extract of the Azure Pipeline YAML that is being used.

parameters:
  dependsOn: ''
  vmImage: ubuntu-latest
  environment: test
  project: api
  newmanFile: api-rest-v1.postman.json
stages:
  - stage: run_portman_tests
    displayName: Run Portman tests
    dependsOn: '${{ parameters.dependsOn }}'
    jobs:
      - job:
        displayName: 'Execute Portman test'
        steps:
          - checkout: self
            displayName: Git checkout
            persistCredentials: false
            submodules: true

          - task: NodeTool@0
            displayName: Use Node 16.x
            inputs:
              versionSpec: 16.x

          - bash: npm ci
            workingDirectory: $(Build.SourcesDirectory)/testing/newman
            displayName: NPM ci

          - bash: npm run portman
            workingDirectory: $(Build.SourcesDirectory)/testing/newman
            displayName: 'Run portman'

          - bash: newman run ${{ parameters.newmanFile }} -e ./config/envs/test.env.json -r htmlextra --reporter-htmlextra-export newman-${{ parameters.project}}-report.html --reporters=junit,cli --reporter-junit-export='newman-${{ parameters.project}}-report.xml' --insecure
            workingDirectory: $(Build.SourcesDirectory)/testing/newman
            displayName: Run newman tests for "${{ parameters.project }}" - env  ${{ parameters.environment }}

          - task: PublishTestResults@1
            displayName: Publish Newman Test Results for "${{ parameters.project }}" -  env ${{ parameters.environment }}
            condition: succeededOrFailed()
            inputs:
              testResultsFiles: $(Build.SourcesDirectory)/testing/newman/newman-*.xml
              testRunTitle: Newman test suite "${{ parameters.project }}" - env ${{ parameters.environment }}

          - task: PublishPipelineArtifact@1
            displayName: Publish Newman artifacts
            condition: succeededOrFailed()
            inputs:
              targetPath: $(Build.SourcesDirectory)/testing/newman
              artifactName: ${{ parameters.project }}-newman-${{ parameters.environment  }}-$(System.JobAttempt)

[shubhambajad]

@thim81 Thank you for your response it is really helpful I will try to configure it.