Jenkins X Pipelines Internals Part 2 — Meta Pipeline

This is the second part of a series of blog posts on the internals of the Jenkins X Pipelines. We’ll dive into the “Meta Pipeline”: the part of the workflow which is responsible for retrieving the right pipeline to execute, and to translate it into a format that Tekton can understand.

In the first part of this series, we’ve walked through everything that happened in the cluster, from the incoming GitHub WebHook event to a running Tekton pipeline. But pipelines can be complex, and built from multiple levels of inheritance. This is the case with the Jenkins X Pipelines, which supports Build Packs to abstract away most of the complexity for the developers. The downside is that building the whole pipeline is now more complex because it requires retrieving parts of it from multiple git repositories — before combining them all together. Doing all that work is too much for the Jenkins X PipelineRunner Controller, which is why this part has been extracted… into a pipeline of its own. The “meta pipeline”.

So Jenkins X will in fact run 2 pipelines each time it needs to run your pipeline:

• the meta pipeline first, whose job is to build your pipeline first, and then translate it into a format that Tekton can understand
• and then your pipeline

You can see it by running the jx get activities command for example:

Output of the `jx get activities` command

The first block of steps is the meta pipeline, which is a pipeline of its own. And then the second block of steps is coming from our real pipeline.

The meta pipeline is generated by the Jenkins X PipelineRunner Controller, and its definition is coded in the metapipeline.go file. It has the following steps:

Workspace setup

First, it will ensure that it has checkout the right version of your git repository, by running the jx step git merge command. When Jenkins X is running a pipeline for your Pull Request, it won’t just use the PR’s branch. Because your branch might be based on an old commit from the master branch, and even if it builds in isolation, it might not work when integrated with the latest changes from the master branch. This is Continuous Integration, after all, the goal is to see the result of integrating all the work together.

This is why Jenkins X will always run the Pull Request Pipelines on top of a local merge of your branch with master. To do that, it will first check-out the master branch, and then merge your branch’s commit.

Showing logs for build githubOrg/repoName/pr-1 #1 pr-build stage meta-pipeline and container step-merge-pull-refs
DEBUG: ran git fetch origin c724a5e3964eb510ab988e61e0b6abc82976743a: d0cb65ef54f0a80a64dfc0e3c532eb55279b4d24: in
DEBUG: Unshallowed git repo in
DEBUG: ran git checkout master in
DEBUG: ran git reset --hard d0cb65ef54f0a80a64dfc0e3c532eb55279b4d24 in
DEBUG: ran clean --force -d . in
DEBUG: ran git merge c724a5e3964eb510ab988e61e0b6abc82976743a in
Merged SHA 3abe373e5476221fda4c93b51a3b40ebe0b038ff with commit message 'Merge commit 'c724a5e3964eb510ab988e61e0b6abc82976743a'' into base branch master
Merged SHA c724a5e3964eb510ab988e61e0b6abc82976743a with commit message 'add feature' into base branch master
Merged SHA 759626ae55f40531508f8070599ac9e392b24ee7 with commit message 'fix something' into base branch master

Note that the git clone operation has already been done in a prior step — we’ll see that in another blog post.

Effective Pipeline

Now that the workspace is set up as it should, it is time to build the “effective pipeline”. This is the fully-parsed representation of your pipeline, with all the steps coming from build packs and its inheritance mechanism.

For example, your jenkins-x.yml file could be as simple as:

buildPack: go

This means that you rely on the go “build pack”, which itself extends the “classic” go build pack.

To build the effective pipeline, Jenkins X will have to clone both the Classic & Kubernetes build packs git repositories, and then combine the pipeline parts together. This is done by the jx step syntax effective command. You can run the same command on your laptop. The result will be something like:

pipelineConfig:
  agent: [...]
  pipelines:
    pullRequest:
      pipeline:
        options: [...]
        stages: [...]
    release:
      pipeline:
        options: [...]
        stages: [...]
      setVersion: [...]

Tekton Resources Generation

The last step of the meta pipeline is to translate your pipeline from its Jenkins X representation into the Tekton representation. In other words, it needs to generate the Tekton resources: Pipeline, Tasks, and PipelineRun. This is done by the jx step create task command. It is the same logic that was used in the Jenkins X PipelineRunner Controller (see part 1) to generate the meta pipeline because it is the same code.

The first thing it will do is generate the version number that will be available as an input parameter for your pipeline. This is not the build number — which has already been generated prior to running the meta pipeline — but your application’s next version. It has different ways to generate that version:

• for a Pull Request Pipeline, it will just use the 0.0.0-SNAPSHOT-PR-${PR_NUMBER}-${BUILD_NUMBER} template.
• for a Release Pipeline, it will either use the user-provided step or fallback to the jx step next-version command — which will find the next version by incrementing the last tag. More on that in a later blog post.

The version will be set as a pipeline parameter named version, and the build number as a parameter named build_id.

It will then iterate over all your stages, and convert them into Tekton Tasks. This is done in the stageToTask function. An interesting thing to note here is that (almost) all the tasks created will have a first step named git-merge. This step runs the jx step git merge command and is here to ensure your workspace is set up with the right content — same as what has been done at the beginning of the meta pipeline. So both pipelines use the same command to “checkout” the workspace.

It will also create a Tekton Pipeline to group the tasks together, and a Tekton PipelineRun to ask Tekton to run the pipeline.

It will then retrieve the Jenkins X PipelineActivity that has been created by the Jenkins X PipelineRunner Controller, while preparing the meta pipeline. This resource is used by Jenkins X as the entry point to interact with all the Tekton resources (pipelines, pipeline runs, tasks, …) associated with a Jenkins X Pipeline. The PipelineActivity was already associated with the meta pipeline resources, and it will now be associated with your pipeline resources. So when you run commands such as jx get activities or jx get build logs, it will use the PipelineActivity.

Another use of the PipelineActitivty is the automatic deletion of all the Tekton resources created by Jenkins X. Jenkins X has a few Garbage Collector running in the cluster, including one named jenkins-x-gcactivities, which is used to delete the PipelineActivities once they are not needed anymore. And because the PipelineActivity has been set as the “owner” of the Tekton resources, when it will be deleted, Kubernetes will also delete its “children” in cascade.

The jx step create task command that is used inside the meta pipeline to generate the Tekton resources can also be run locally if you are curious to see how your Jenkins X Pipeline would be translated into a Tekton Pipeline:

• first, you need to run the jx step syntax effective --output-dir=. command to generate a jenkins-x-effective.yml file
• then you can run the jx step create task --kind pullrequest --dry-run --view command, that will read the jenkins-x-effective.yml file and translate it in Tekton format. The --view flag will display a nice-looking table of all the steps of your pipeline:

NAME       COMMAND                     IMAGE
git-merge  jx step git merge --verbose gcr.io/jenkinsxio/builder-jx:2.0.1117-453
unit-tests /bin/sh -c make test        golang:1.13
build      /bin/sh -c make build       golang:1.13

• you can also run it without the --view flag, such as jx step create task --kind pullrequest --dry-run and it will write the Tekton resources in the out directory. We’ll explore the content of these resources in the next blog post.

The jx step create task command is full of options, including the --interpret one, which is used to run the pipeline directly. As you can see in the source code, it will run the commands for each step directly on your laptop, without using any container or abstraction. This is mainly used to run the initial jx boot pipeline on your laptop, but it can also be used to run any pipeline — as long as the steps commands can run locally.

As we saw in the previous blog post, when we’ll create the Tekton PipelineRun, Tekton will pick it up and create the associated pods to run the pipeline — which is your pipeline. So at the end of the meta-pipeline, your pipeline will automatically be started.

In the next blog post, we’ll focus on the stages that compose a pipeline, and we’ll see how they are implemented using Tekton.