Introducing Octopilot: a CLI to automate the creation of GitHub pull requests in your gitops workflow
Gitops is often defined as “operations by pull request”. But how do you create these pull requests? With Octopilot!
When we started practicing gitops at work, we used updatebot to automate the creation of pull requests on the git repositories that represents our environments. Why updatebot? Because it was part of the Jenkins X project, and we’re using Jenkins X for our Continuous Delivery platform.
But very soon we wanted to use our — successful — gitops workflow for more projects, and we felt limited by the tool. So we started writing our own CLI tool to automate the creation of pull requests: Octopilot. This was more than one year ago. Although the codebase was open-source from day one, we didn’t really communicate on it — we just used it for our own needs. We’ve been using it with great success so far, creating thousands of pull requests — and merging more than half of these. But recently we spend time polishing it, writing documentation, and releasing a v1. So it’s time to introduce Octopilot to the world!
What is Octopilot?
Octopilot is a CLI tool designed to help you automate your gitops workflow, by automatically creating and merging GitHub pull requests to update specific content in git repositories. Note that the “GitHub” part of this sentence is important: Octopilot only works with GitHub-hosted git repositories — as you can say from the name (“octo”…)
We see Octopilot as a swiss army knife to propagate changes in a gitops-based infrastructure. For example, we’re using it to:
- promote new releases of our applications to the staging and production environments
- promote new releases of our libraries to all the applications using them
- synchronize certificates managed by cert-manager to multiple clusters — through git of course
- and more!
Let’s get technical! With Octopilot, you can:
- manipulate YAML or JSON files as you want, using the powerful expression syntax from the great yq CLI — inspired by the famous jq CLI. Because, yes, most configuration files in the gitops world are YAML files.
- update files encrypted with sops — because who wants to store non-encrypted sensitive data in git?
- update files with regex-based rules — for these times when you need raw power.
- or even execute any command/tool — because you don’t want to be limited by what we support.
How does it work?
- you give it one or more repositories to work on, and Octopilot will clone them.
- for each one, it will run all your “update rules” — such as replacing a version in a configuration file, executing an external program, …
- if there are changes, it will commit them, and create a pull request — or update an existing pull request with the same set of labels. Note that there are 3 different strategies for updating pull requests: appending new commits, resetting the pull request from the base branch, or just create a new pull request every time.
- optionally, Octopilot can also merge the pull requests — once all your Continuous Integration pipelines have been successfully executed of course!
What does it look like to use Octopilot?
We use CLI flags to configure Octopilot’s behavior: which repositories to clone, the update rules, the pull request settings, …
We’ve designed Octopilot to be used in a Continuous Delivery pipeline:
- it has no dependencies — not even
git - it doesn’t need configuration files
- it can update multiple repositories with a single command
This makes it very easy to run in a CD pipeline, such as Jenkins, Jenkins X, Tekton, GitHub Actions, … At work, we’re using it through Jenkins X/Tekton pipelines — and also a few Jenkins pipelines — with success.
The importance of the pull request in a gitops workflow
One of the reasons we wrote Octopilot is to customize the pull requests: title/body of course, but also how they are created, updated, and merged. Because we believe that the pull request has a central role in a gitops workflow: it is the link between multiple repositories, and if neglected it can be hard for people to understand the workflow. But if it’s done right, it will be easy for people to understand how the gitops workflow is set up, and the impact of their changes.
As an example, if we promote a new application release, we should include the release notes in the promotion pull request, such as:
Note that the application’s pull request(s) are linked in the promotion pull request(s), which allow GitHub to generate back-links in the application’s pull request:
This way, a developer submitting a pull request on an application can see how his changes are propagated after the pull request is merged:
- a new release is automatically created
- a promotion pull request has been automatically created on the staging environment git repository and merged by Octopilot, thus enabling Continuous Deployment in the staging environment
- a promotion pull request has been automatically created on the production environment git repository. This promotion PR has been manually merged later.
Everything is stored in git, and GitHub provides links to navigate between all the pieces — connecting the dots.
Why Octopilot?
The main reason we started developing Octopilot is that there was no existing software that matches our needs: integration with sops, excellent support for YAML files — to avoid complex regex rules — and the ability to customize how the pull requests are created and updated.
We wrote Octopilot in Golang because it’s an easy way to embed support for sops without requiring people to install it first — as it’s written in Go too — and because there are excellent Git or YAML libraries — such as go-git and YQ. Coupled with Golang’s ability to create static binaries, and the fact that we’re already using Golang at work, makes it a perfect match for our requirements.
Note that these days there are more tools to automate the creation of pull requests for your gitops workflows:
- jx-updatebot: a Golang-rewrite of the original Java-based updatebot. Used in Jenkins X to propagate changes to git repositories.
- updatecli: another Golang-based CLI. Used in the Jenkins infrastructure.
Each tool has its own specific features.
What’s next?
If you’re interested in testing Octopilot, or if you want to know more about its features, you can read its documentation. We have a section on our use-cases, where we explain what we are doing, why, and how we are using Octopilot to achieve our goals.
As for the future of Octopilot, we have a few ideas, such as supporting one-off “campaigns” with a nice UI — for example for these times when you want to update Go to the latest version on all your Go-based applications/repositories, and you need to preview the changes first, and then to follow the pull requests to see which ones weren’t automatically merged because of CI pipelines failures.
