Jenkins X Pipelines Internals Part 1 — From GitHub WebHook Event to Tekton Pipeline

This is the first part of a series of blog posts on the internals of the Jenkins X Pipelines. It is a walk-through of everything that happens in the cluster, starting from a GitHub WebHook event until we’ll have a running Tekton pipeline.

8 min readFeb 3, 2020

The “new” Jenkins X Pipelines — starting from Jenkins X 2.0 — are implemented using open-source components such as Prow and Tekton. In a few words, Prow has been built into and for Kubernetes, and is responsible for all the GitHub-related automation: reacting to GitHub events, triggering Jobs to run tests, automatically merging Pull Requests, Chat-Ops style user-interface, and so on. It has been battle-tested by the Kubernetes community, which is one of the biggest open-source communities out there. And Tekton is a young project extracted from the Knative project, designed to provide the building blocks for running Pipelines on Kubernetes. More and more projects are starting to build their pipeline systems on top of Tekton, with Jenkins X and OpenShift being the 2 major open-source ones. See Tekton’s “friends” repository for a list of projects built on top of Tekton.

In this blog post, we’ll dive into the internals of Prow, Tekton and Jenkins X, to understand what happens when a GitHub WebHook event is triggered. As you’ll see, it’s clearly not trivial. But don’t worry, you’ll not need to know all that to run your pipeline. It’s just meant to satisfy your curiosity ;-)

Initial configuration

Everything starts when you run the jx import command. We won’t go into the details of what happens when you run this command here, this will be in a later blog post. For the moment, you only need to know that this will create a new Kubernetes resource using the SourceRepository Custom Resource Definition (CRD) defined by Jenkins X. You can see an example here. This will be used to configure Prow so that it will know how to react when it will receive a GitHub WebHook Event for the GitHub repository defined in the SourceRepository. It will also be used to register a new WebHook on the said repository, to notify Prow every time something happens for your repository.

Prow and Hook

Now, you will create a new Pull Request on your GitHub repository with a very important change. GitHub will send an HTTP request to Prow, using the URL defined in your repository’s WebHooks settings.

In fact, Prow is composed of multiple components, and the one that will receive your WebHook event is called… hook. If you list the deployments or pods running in your Jenkins X Kubernetes cluster, in the jx namespace, you can see a deployment called hook. It is coming from the Prow Chart maintained by the Jenkins X team and is using Jenkins X’s fork of Prow. If you are wondering about the strange repository name (test-infra), it is because Prow was developed by the Kubernetes Test Infrastructure team.

So our hook pod is started using the hook command and is listening on the /hook endpoint for GitHub WebHook events. When Hook receives such an event, it will dispatch it to one or more plugins. Plugins can either be internal Prow plugins, or external plugins. External plugins are just HTTP servers to which Hook will send an HTTP request. Internal plugins are registered in Prow’s source code and running in hook’s process. Plugins are configured in Prow’s configuration, which is defined in a Kubernetes ConfigMap named plugins, and is auto-generated by Jenkins X when you run the jx import command for example. More on that in a later blog. You can run kubectl get configmap plugins -n jx to see this configuration.

plugins:
  githubOrg/repoName:
  - approve
  - trigger
  [...]
triggers:
- repos:
  - githubOrg/repoName
  trusted_org: githubOrg

The plugin we care about right now is named trigger, and is responsible for triggering one or more “jobs”. It starts by retrieving the matching “pre-submit” jobs from its configuration, which is stored in the config ConfigMap, and also auto-generated by Jenkins X. Here is a sample of that configuration:

presubmits:
  githubOrg/repoName:
  - agent: tekton
    always_run: true
    context: pr-build
    name: pr-build
    rerun_command: /test this
    trigger: (?m)^/test( all| this),?(\s+|$)

Prow has 2 kinds of job:

presubmit: this is a job executed in the context of a Pull Request before the branch is merged in master.
postsubmit: this is a job executed on the master branch, once the Pull Request has been merged. We’ll talk more about post-submit jobs and “release” pipelines in a later blog post.

The trigger plugin will then create a Kubernetes resource using the ProwJob CRD defined by Prow, for each pre-submit job defined in the configuration:

apiVersion: prow.k8s.io/v1
kind: ProwJob
metadata:
  name: 00bd20bf-394e-11ea-ad56-7e495e75e966
  labels:
    prow.k8s.io/job: pr-build
    prow.k8s.io/refs.org: githubOrg
    prow.k8s.io/refs.pull: "1"
    prow.k8s.io/refs.repo: repoName
    prow.k8s.io/type: presubmit
spec:
  agent: tekton
  context: pr-build
  job: pr-build
  refs:
    org: githubOrg
    pulls:
    - number: 1
      [...]
    repo: repoName
    repo_link: https://github.com/githubOrg/repoName
    [...]
  type: presubmit

As you can see, it contains all the information required to run your pipeline (I skipped some of the fields to keep the sample concise). The important thing to note is the agent field, which is set to tekton. This value is coming from the configuration generated by Jenkins X. Prow’s default agent is kubernetes, but Jenkins X override it, to be able to map this job to a Tekton pipeline using its own custom logic.

The pipeline controllers

If you are not familiar with the notion of “controller” in Kubernetes, it is just a daemon watching for specific events — such as the creation of a resource — and reacting to them. A classic example is the ReplicaSet Controller, which adjusts the number of pods running based on the required number of replicas defined in a ReplicaSet resource. It is a very classic pattern in Kubernetes world, and each time you will create a Kubernetes resource — either a built-in one or a CRD — you can expect that there is a controller running somewhere that will react to the change.

In our case, Hook created a ProwJob, and Prow has a “Pipeline” controller running that will react to it. It is running in the pipeline deployment, which is coming from the same Prow Chart maintained by the Jenkins X team. In fact, this controller has been contributed to Prow by the Jenkins X team. It is started using the pipeline command and watches for changes in the ProwJob resources. But in fact, it will only care about ProwJobs defined with the tekton agent. ProwJobs using different agents — such as kubernetes, jenkins or knative-build — will be handled by different controllers.

The Prow Pipeline Controller has 2 modes:

a “direct” one, where it will create the Tekton resources itself if the ProwJob embeds the Tekton Pipeline spec. This is meant for people who want to work directly with the Tekton syntax for example.
a “delegate” one, where it will forward the job request to a different component, which will be responsible for the creation of the Tekton resources. This mode will be used if the ProwJob does not embed the Tekton Pipeline spec.

Jenkins X uses the second mode because its goal is to abstract away the complexity of Tekton, so it needs to use an extra layer to convert your pipeline from the Jenkins X syntax to the Tekton syntax.

Jenkins X PipelineRunner Controller

This is a pure-Jenkins X component, running in the pipelinerunner deployment, and deployed as part of the Prow Chart maintained by the Jenkins X team. It is started using the jx controller pipelinerunner command, and listen for HTTP requests.

It is responsible for parsing the PipelineRunRequest coming from Prow’s Pipeline Controller, retrieving the Git repository to clone, branch/commit to checkout, type of pipeline to run — either pullRequest or release — and the build number, which is just an incremental ID. Note that this build number is stored in the SourceRepository resource for the repository.

Once it has all those pieces of information, you’ll think that it has everything it needs to retrieve your pipeline from Git, translate it in the Tekton syntax, and ask Tekton to run it. Well… not yet. Because Jenkins X Pipelines can use multiple levels of inheritance, cloning all the Git repositories involved and building your “effective” pipeline is too much work for this component. Instead, all the hard work is delegated (again) to … a pipeline. Yes, Jenkins X will run a pipeline before your pipeline, just to build your pipeline. This is called the “meta pipeline”. The second part of this series will be focused on it.

Back to our Pipeline Runner, which will generate all the resources required to run the meta pipeline:

a Tekton Task with a few steps, that we’ll explore in the next blog post.
a Tekton Pipeline to wrap our Task.
a Tekton PipelineRun to start our pipeline.
a Jenkins X PipelineActivity, which is the entry point in Jenkins X to interact with a Pipeline — for example by running the jx get activities command. This resource is set as the “owner” of the Tekton resources so that when it is deleted, all its children will be deleted too.
a Jenkins X PipelineStructure to glue all the Tekton resources together: pipeline/ pipelinerun and task/ taskrun.

Tekton Pipelines Controller

As you would expect by now, Tekton has controllers too, and we’ll focus on 2 here:

pipelinerun controller, which watches for PipelineRun events
taskrun controller, which watches for TaskRun events

Both of them are running in a single process, in the tekton-pipelines-controller deployment, which is part of the Tekton Chart maintained by the Jenkins X team. As you can see in the default values Jenkins X is using a fork of Tekton, with container images coming from gcr.io/abayer-pipeline-crd/tekton-for-jx and its source code is located in the github.com/abayer/build-pipeline repository — more specifically in the 0.8.0-jx-support-backwards-incompats branch. All that may change in the future, of course ;-)

So the Jenkins X PipelineRunner Controller created a few Tekton resources, including a PipelineRun resource. This resulted in Tekton’s Pipelinerun Controller handling our new PipelineRun resource and creating a TaskRun resource for each Task in the Pipeline referenced by the PipelineRun.

Tekton’s Taskrun Controller will then react to the new TaskRun resources and will create a Pod for each one. This pod has all the containers needed to execute all the steps of the meta pipeline — as we’ll see in more detail in the next blog post.

The big picture

In summary, the workflow is:

GitHub sends an HTTP request to Hook, with a WebHook Event JSON payload
Hook delegates the event to the trigger plugin, which creates one or more ProwJob resources
Prow’s Pipeline Controller watches the ProwJob creation, and forwards it to the Jenkins X PipelineRunner Controller, using an HTTP request
Jenkins X’s PipelineRunner Controller creates all the required Tekton resources — Pipeline, PipelineRun, and Tasks
Tekton’s Pipelinerun Controller watches the PipelineRun creation, and create one TaskRun for each of the tasks referenced by the pipeline
Tekton’s Taskrun Controller watches the TaskRun creation, and create a Pod for each Task

Easy, isn’t it? ;-)

In the next blog post, we’ll dive into the “Meta Pipeline”: the part of the workflow which is responsible for retrieving the right pipeline to execute, and to translate it into a format that Tekton can understand.