Jenkins X Pipelines Internals Part 4— Steps

This is the fourth part of a series of blog posts on the internals of the Jenkins X Pipelines. We’ll focus on the “steps” that compose a pipeline, and we’ll see how they are implemented using Tekton.

In the previous part of this series, we’ve walked through the internals of the stages that compose a pipeline. Now we’ll see how the Jenkins X Steps — that compose a stage — are implemented.

First, a few pointers:

• the reference documentation for the steps
• the source code definition of a step

These steps — we’ll call them Jenkins X Steps — are converted by the meta pipeline into Tekton Steps. And a Tekton Step is just a Kubernetes Container, as you can see in the source code definition of a Tekton Step.

So:

• Jenkins X Stages are converted into Tekton Tasks — which are converted into Kubernetes Pods
• Jenkins X Steps are converted into Tekton Steps in the generateSteps function (in Jenkins X). Each Tekton step is then converted into a Kubernetes Container in the MakePod function in the Tekton taskrun Controller.

But steps are meant to be run sequentially in a stage/task, and if you have a bit of experience with Kubernetes, you know that all containers in a pod run in parallel. So what’s the magic trick to make them run sequentially?

In the beginning, Tekton — or its predecessor, Knative — used only init-containers. Init containers are specific containers in a pod: they are run before the “main” containers, and one after the other. Usually, they run small scripts with the aim of initializing the pod environment — for example by populating a shared volume. But init-containers have limitations, as explained in a Tekton issue — a big one being the absence of Kubernetes native log persistence. So Tekton switched to “classic” containers to solve these limitations — you can see the pull request for all the details.

Let’s see what happens if we run the following pipeline, with 2 simple steps:

buildPack: none
pipelineConfig:
  pipelines:
    pullRequest:
      pipeline:
        stages:
          - name: test-and-build
            steps:
              - name: unit-tests
                command: make test
                image: golang:1.13
              - name: build
                command: make build
                image: golang:1.13

If we retrieve the associated pod with the jx get build pods -r yourGithubRepo command and then inspect it with kubectl get pod githubOrg-repoName-pr-1p5s4fp-1-test-and-build-7p5lx-pod-868290, we can see a few containers. Let’s start with the first one:

name: step-git-source-githubOrg-repoName-pr-71-nwf4f
image: gcr.io/abayer-pipeline-crd/tekton-for-jx/git-init:0.8.0-for-jx
command:
- /builder/tools/entrypoint
args:
- -wait_file
- /builder/downward/ready
- -post_file
- /builder/tools/0
- -wait_file_content
- -entrypoint
- /ko-app/git-init
- --
- -url
- https://github.com/githubOrg/repoName.git
- -revision
- master
- -path
- /workspace/source
workingDir: /workspace
env:
- name: HOME
  value: /builder/home
volumeMounts:
- mountPath: /builder/tools
  name: tools
- mountPath: /builder/downward
  name: downward
- mountPath: /workspace
  name: workspace
- mountPath: /builder/home
  name: home

It is running a command called entrypoint. As we can see in the documentation, it is used to wrap the real command to execute and ensure that the steps will be executed in order. The transformation of the step’s original command is done in the Tekton taskrun Controller, in the RedirectStep function.

It will first wait for the file /builder/downward/ready to be available. This file is in the downward volume, which is defined as:

volumes:
  - name: downward
    downwardAPI:
      defaultMode: 420
      items:
        - path: ready
          fieldRef:
            apiVersion: v1
            fieldPath: metadata.annotations['tekton.dev/ready']

The Kubernetes Downward API is used to expose Kubernetes-related information to containers. In our case, the ready file will contain the value of a pod annotation. This annotation will be added by the Tekton taskrun Controller once the pod is ready, and the TaskRun status has been updated.

So the container will start at the pod creation — as usual — running the entrypoint command, but waiting for a specific annotation to be present. Then it will run the command provided through the -entrypoint flag. In our case, this is the git-init command, used to clone the Git repository. When this command will finish, it will create a file at /builder/tools/0 and then exit.

Note that the pod has restartPolicy: Never — which means that any container exiting won’t be restarted.

The second container has the following command/args:

command:
- /builder/tools/entrypoint
args:
- -wait_file
- /builder/tools/0
- -post_file
- /builder/tools/1
- -entrypoint
- jx
- --
- step
- git
- merge
- --verbose

So it will first wait for the creation of the file /builder/tools/0 — meaning for the end of the first container. And then run its own command, and finish by creating a file at /builder/tools/1. This will then trigger the next container/step. And so on.

But what is this entrypoint command? Where is it coming from? It is stored in the tools volume, which is defined as an emptyDir volume:

volumes:
  - name: tools
    emptyDir: {}

This volume is populated by an init container, which just copies the entrypoint binary from its container image into the shared volume:

initContainers:
  - name: step-place-tools
    image: gcr.io/abayer-pipeline-crd/tekton-for-jx/entrypoint:0.8.0-for-jx
    command:
      - /bin/sh
    args:
      - -c
      - cp /ko-app/entrypoint /builder/tools/entrypoint
    volumeMounts:
      - mountPath: /builder/tools
        name: tools

This init-container is inserted in the pod by the Tekton taskrun Controller, in the createdRedirectedTaskSpec function. This function is also responsible for the definition of the related tools volume, and calling the RedirectSteps function which will replace the command for each container.

Environment variables

If you inspect the environment variables made available to your containers, you can see that there are a lot of them:

env:
- name: HOME
  value: /builder/home
- name: APP_NAME
  value: repoName
- name: BRANCH_NAME
  value: PR-1
- name: BUILD_ID
- name: BUILD_NUMBER
  value: "1"
- name: JOB_NAME
  value: githubOrg/repoName/PR-1
- name: JOB_SPEC
  value: type:presubmit
- name: JOB_TYPE
  value: presubmit
- name: PIPELINE_CONTEXT
  value: test-and-build
- name: PIPELINE_KIND
  value: pullrequest
- name: PROW_JOB_ID
- name: PULL_BASE_REF
  value: master
- name: PULL_BASE_SHA
  value: 5752a1e3611481066c9570e39b5d9acf51267680
- name: PULL_NUMBER
  value: "1"
- name: PULL_PULL_SHA
  value: 3446577e200f5cdb5859e1b434b660ec80f7ccba
- name: PULL_REFS
  value: master:5752a1e3611481066c9570e39b5d9acf51267680,1:3446577e200f5cdb5859e1b434b660ec80f7ccba
- name: REPO_NAME
  value: repoName
- name: REPO_OWNER
  value: githubOrg
- name: SOURCE_URL
  value: https://github.com/githubOrg/repoName.git
- name: DOCKER_REGISTRY
- name: GIT_AUTHOR_NAME
  value: jenkins-x-bot
- name: GIT_AUTHOR_EMAIL
  value: jenkins-x@googlegroups.com
- name: GIT_COMMITTER_NAME
  value: jenkins-x-bot
- name: GIT_COMMITTER_EMAIL
  value: jenkins-x@googlegroups.com
- name: JX_BATCH_MODE
  value: "true"
- name: VERSION
  value: 0.0.0-SNAPSHOT-PR-1-1
- name: PREVIEW_VERSION
  value: 0.0.0-SNAPSHOT-PR-1-1

This list of environment variables is built inside the meta pipeline by the buildEnvParams function.

Container Image and Jenkins X Version Stream

You might have noticed that sometimes Jenkins X Pipelines references container images without explicit versions — such as Jenkins X’s own pipeline — but in the resulting pod, the container image has a specific version set.

So the following Jenkins X Step:

steps:
  - name: do-something
    image: gcr.io/jenkinsxio/builder-go
    ...

is converted into the following container:

containers:
  - name: do-something
    image: gcr.io/jenkinsxio/builder-go:2.0.1155-487
    ...

This version is coming from Jenkins X Version Stream — which is basically a Git repository storing versions to use for different components. The version of our builder-go image is stored in the docker/gcr.io/jenkinsxio/builder-go.yml file.

The image transformation is done as part of the steps conversion — from Jenkins X Steps to Tekton Steps — using the ResolveDockerImage function.