How to protect a Kubernetes Ingress behind Okta, with Nginx

In this blog post, we’ll see how easy it is to “protect” a web app behind Okta, using Nginx as a reverse-proxy in front of it. In a Kubernetes environment.

Our use-case

We’re using both Kubernetes to deploy our applications and Okta as a company SSO. We’re also big fans of Jenkins X. Jenkins X comes with a few UI, which unfortunately don’t have native authentication/authorization implementation yet:

• CloudBees Jenkins X UI
• Deck — Prow’s dashboard
• and we also deployed Tekton Dashboard

Jenkins X relies on Nginx for its ingress controller, and it uses the basic auth feature to protect its UI by default. The issue with this solution is that you either need to manually manage all your users (and passwords), or give them a shared set of credentials.

As we’re already using Okta at work, we wanted to integrate it into our Jenkins X setup.

Nginx, Vouch, and Okta

We found a blog post written by the Okta team on how to Use Nginx to Add Authentication to Any Application, which is already quite good. I won’t repeat everything they wrote, but instead, I’ll focus on the steps you’ll have to do if you’re working in a Kubernetes environment.

There are 3 parts in this setup:

• create new applications in your Okta account
• install and configure Vouch Proxy
• configure the ingresses you want to protect

Okta Application(s)

First, you’ll need to create a new OpenID (Web) Application in Okta, with the “login redirect URL” set to something such as https://vouch.jx.example.com/auth — which is the Vouch Proxy Auth endpoint. Make sure to set the “grant type allowed” to Authorization Code. And keep the application’s client_id and client_secret, we’ll need them later.

All the web-apps you want to protect will redirect the users to the Vouch Proxy Login endpoint, which will then redirect to Okta using the OpenID Application you just created. So you can have as many web-apps as you want behind this single Okta Application. But the issue is that your users won’t see their web-apps in their Okta Dashboard.

The “solution” we found is to create a fake SWA Application for each web-app, with empty credentials. The SWA Application is only here to get a link to the real web-app from the Okta Dashboard.

Vouch Proxy

The vouch-proxy’s README is well written, and it contains a small section about how to run it in a Kubernetes cluster — with a link to a Helm chart.

We’re big fans of Helmfile — a tool used to manage your Helm releases in Gitops style. Installing Vouch in a Helmfile-managed cluster is as simple as adding to your helmfile.yaml:

repositories:
  - name: vouch
    url: https://halkeye.github.io/helm-charts
releases:
  - name: vouch
    chart: vouch/vouch
    version: "0.1.4"
    namespace: jx
    values:
      - vouch/values.yaml
    secrets:
      - vouch/secrets.yaml

and then you can customize Vouch’s configuration in a vouch/values.yaml file:

replicaCount: 1
ingress:
  enabled: true
  annotations:
    kubernetes.io/ingress.class: nginx
  hosts:
    - vouch.jx.example.com
  paths: ["/"]
  tls:
    - secretName: tls-jx-example.com-p
      hosts:
        - vouch.jx.example.com
config:
  vouch:
    allowAllUsers: true
    cookie:
      name: jxVouchCookie
      domain: jx.example.com
      secure: true
      httpOnly: true
      maxAge: 14400
    jwt:
      issuer: Vouch
      maxAge: 240
      compress: true
    session:
      name: jxVouchSession
  oauth:
    provider: oidc
    auth_url: https://yourdomain.okta.com/oauth2/v1/authorize
    token_url: https://yourdomain.okta.com/oauth2/v1/token
    user_info_url: https://yourdomain.okta.com/oauth2/v1/userinfo
    scopes:
      - openid
      - email
      - profile
    callback_url: https://vouch.jx.example.com/auth

Note that Okta’s blog post uses URLs such as https://yourdomain.okta.com/oauth2/default/v1/... — but we found that it was not working in our case, because we don’t use Okta’s API Access Management product. The fix is to use https://yourdomain.okta.com/oauth2/v1/... URLs instead, as documented in Okta’s Help Center — or in Okta’s OpenID Connect Reference Documentation about URLs.

The client_id / client_secret of your Okta application are stored in the vouch/secrets.yaml file, which is a sops-encrypted file. Helmfile has native support for the helm-secrets plugin, which uses sops to encrypt/decrypt YAML files. As our Kubernetes clusters are hosted on GCP, we’ve configured sops to use GCP KMS API to get encryption/decryption keys. This way we can control who has permission to encrypt/decrypt our secrets, and we’re writing them in our git repositories, along with the other settings — Gitops style.

In our case, we configured Vouch to run at vouch.jx.example.com and to store the cookie on the jx.example.com domain. Which means that this instance will be able to protect ingresses exposed under the same domain, such as jxui.jx.example.com or deck.jx.example.com.

Ingress configuration

Protecting an existing web-app that is already exposed through an ingress requires just a few annotations — as long as this ingress is exposed using the ingress-nginx controller. Here is an example:

kind: Ingress
apiVersion: extensions/v1beta1
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/auth-response-headers: X-Vouch-User
    nginx.ingress.kubernetes.io/auth-signin: https://vouch.jx.example.com/login?url=$scheme://$http_host$request_uri&vouch-failcount=$auth_resp_failcount&X-Vouch-Token=$auth_resp_jwt&error=$auth_resp_err
    nginx.ingress.kubernetes.io/auth-snippet: |
      auth_request_set $auth_resp_jwt $upstream_http_x_vouch_jwt;
      auth_request_set $auth_resp_err $upstream_http_x_vouch_err;
      auth_request_set $auth_resp_failcount $upstream_http_x_vouch_failcount;
    nginx.ingress.kubernetes.io/auth-url: https://vouch.jx.example.com/validate

You can see all the supported annotations in the documentation. The Nginx Ingress Controller will do the hard work on generating the right Nginx configuration to delegate the authentication to Vouch, which itself will talk to Okta.

Expose Controller

Jenkins X doesn’t provide Ingresses directly, instead, it relies on a tool named Expose Controller to automatically create ingresses from services, based on annotations on the services.

In this case, you’ll need to wrap the Nginx-related annotations in the fabric8.io/ingress.annotations annotation:

apiVersion: v1
kind: Service
metadata:
  annotations:
    fabric8.io/expose: "true"
    fabric8.io/ingress.annotations: |-
      kubernetes.io/ingress.class: nginx
      nginx.ingress.kubernetes.io/auth-signin: "https://vouch.jx.example.com/login?url=$scheme://$http_host$request_uri&vouch-failcount=$auth_resp_failcount&X-Vouch-Token=$auth_resp_jwt&error=$auth_resp_err"
      nginx.ingress.kubernetes.io/auth-url: https://vouch.jx.example.com/validate
      nginx.ingress.kubernetes.io/auth-response-headers: X-Vouch-User
      nginx.ingress.kubernetes.io/auth-snippet: auth_request_set $auth_resp_jwt $upstream_http_x_vouch_jwt; auth_request_set $auth_resp_err $upstream_http_x_vouch_err; auth_request_set $auth_resp_failcount $upstream_http_x_vouch_failcount;

An alternative is to configure Nginx’s Global External Authentication and then disable it for the ingresses you don’t want to protect.

When should you use it?

• when you’re already using Nginx as a reverse-proxy / ingress controller
• when you’re already using Okta or another OAuth2 / OpenID Connect provider — supported by Vouch Proxy
• when the web app you want to protect has no native authentication/authorization support
• when you don’t care about who is connected — unless the web app supports reading that information from an HTTP Header
• when you don’t care about logout — because most web app without native authentication/authorization support have no “logout” button

In our case — quickly protect the Jenkins X UIs — it’s a good solution. Mainly because these apps are read-only, and once we know people have been authorized to access the app, we don’t really care about who they are or what they do. Of course with a read-write application, it would be different. This is why our long-term solution is still to wait for Jenkins X to natively implement a real solution ;-)