Rate-limiting and protecting an API with JSON Web Tokens (JWTs) and Kubernetes authnz using Kuadrant

Example of rate-limiting and protecting an API (the Toy Store API) with authentication based on ID tokens (signed JWTs) issued by an OpenId Connect (OIDC) server (Keycloak) and alternative Kubernetes Service Account tokens, and authorization based on Kubernetes RBAC, with permissions (bindings) stored as Kubernetes Roles and RoleBindings.

Pre-requisites

Docker
kubectl command-line tool
jq

Run the guide ❶ → ❼

❶ Setup the environment

Clone the project:

git clone https://github.com/Kuadrant/kuadrant-operator && cd kuadrant-operator

Spin-up the cluster with all dependencies installed:

make local-env-setup deploy

🤔 What exactly does the step above do?

Creates a containerized Kuberentes server using Kind
Installs Istio
Installs Kuberentes Gateway API
Installs the Kuadrant system (CRDs and operators)

❷ Deploy the API

Deploy the application in the default namespace:

kubectl apply -f examples/toystore/toystore.yaml

Create the HTTPRoute:

kubectl apply -f examples/toystore/httproute.yaml

Expose the API:

kubectl port-forward -n istio-system service/istio-ingressgateway 9080:80 2>&1 >/dev/null &

API lifecycle

Lifecycle

Try the API unprotected

curl -H 'Host: api.toystore.com' http://localhost:9080/toy -i
# HTTP/1.1 200 OK

❸ Request the Kuadrant instance

kubectl apply -f -<<EOF
apiVersion: kuadrant.io/v1beta1
kind: Kuadrant
metadata:
  name: kuadrant
spec: {}
EOF

❹ Deploy Keycloak

Create the namesapce:

kubectl create namespace keycloak

Deploy Keycloak:

kubectl apply -n keycloak -f https://raw.githubusercontent.com/Kuadrant/authorino-examples/main/keycloak/keycloak-deploy.yaml

The step above deploys Keycloak with a preconfigured realm and a couple of clients and users created.

The Keycloak server may take a couple minutes to be ready.

❺ Create the `AuthPolicy`

kubectl apply -f -<<EOF
apiVersion: kuadrant.io/v1beta1
kind: AuthPolicy
metadata:
  name: toystore-protection
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: toystore
  authScheme:
    identity:
      - name: keycloak-users
        oidc:
          endpoint: http://keycloak.keycloak.svc.cluster.local:8080/auth/realms/kuadrant
      - name: k8s-service-accounts
        kubernetes:
          audiences:
            - https://kubernetes.default.svc.cluster.local
    authorization:
      - name: k8s-rbac
        kubernetes:
          user:
            valueFrom:
              authJSON: auth.identity.sub
    response:
      - name: rate-limit
        json:
          properties:
            - name: userID
              valueFrom:
                authJSON: auth.identity.sub
        wrapper: envoyDynamicMetadata
        wrapperKey: ext_auth_data
EOF

Try the API missing authentication

curl -H 'Host: api.toystore.com' http://localhost:9080/toy -i
# HTTP/1.1 401 Unauthorized
# www-authenticate: Bearer realm="keycloak-users"
# www-authenticate: Bearer realm="k8s-service-accounts"
# x-ext-auth-reason: {"k8s-service-accounts":"credential not found","keycloak-users":"credential not found"}

Try the API without permission

Obtain an access token with the Keycloak server:

ACCESS_TOKEN=$(kubectl run token --attach --rm --restart=Never -q --image=curlimages/curl -- http://keycloak.keycloak.svc.cluster.local:8080/auth/realms/kuadrant/protocol/openid-connect/token -s -d 'grant_type=password' -d 'client_id=demo' -d 'username=john' -d 'password=p' | jq -r .access_token)

Send requests to the API as the Keycloak-authenticated user (missing permission):

curl -H "Authorization: Bearer $ACCESS_TOKEN" -H 'Host: api.toystore.com' http://localhost:9080/toy -i
# HTTP/1.1 403 Forbidden

Create a Kubernetes Service Account to represent a user belonging to the other source of identities:

kubectl apply -f -<<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: client-app-1
EOF

Obtain an aaccess token for the client-app-1 service account:

SA_TOKEN=$(kubectl create token client-app-1)

Send requests to the API as the service account (missing permission):

curl -H "Authorization: Bearer $SA_TOKEN" -H 'Host: api.toystore.com' http://localhost:9080/toy -i
# HTTP/1.1 403 Forbidden

❻ Grant access to the API

Create the toystore-reader and toystore-writer roles:

kubectl apply -f -<<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: toystore-reader
rules:
- nonResourceURLs: ["/toy*"]
  verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: toystore-writer
rules:
- nonResourceURLs: ["/admin/toy"]
  verbs: ["post", "delete"]
EOF

Add permissions to the users and service accounts:

kubectl apply -f -<<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: toystore-readers
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: toystore-reader
subjects:
- kind: User
  name: $(jq -R -r 'split(".") | .[1] | @base64d | fromjson | .sub' <<< "$ACCESS_TOKEN")
- kind: ServiceAccount
  name: client-app-1
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: toystore-writers
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: toystore-writer
subjects:
- kind: User
  name: $(jq -R -r 'split(".") | .[1] | @base64d | fromjson | .sub' <<< "$ACCESS_TOKEN")
EOF

🤔 Can I use Roles and RoleBindings instead of ClusterRoles and ClusterRoleBindings?

Yes, you can.

The example above is for non-resource URL Kubernetes roles. For using Roles and RoleBindings instead of ClusterRoles and ClusterRoleBindings, thus more flexible resource-based permissions to protect the API, see the spec for Kubernetes SubjectAccessReview authorization in the Authorino docs.

Try the API with permission

Send requests to the API as the Keycloak-authenticated user:

curl -H "Authorization: Bearer $ACCESS_TOKEN" -H 'Host: api.toystore.com' http://localhost:9080/toy -i
# HTTP/1.1 200 OK

curl -H "Authorization: Bearer $ACCESS_TOKEN" -H 'Host: api.toystore.com' -X POST http://localhost:9080/admin/toy -i
# HTTP/1.1 200 OK

Send requests to the API as the service account (missing permission):

curl -H "Authorization: Bearer $SA_TOKEN" -H 'Host: api.toystore.com' http://localhost:9080/toy -i
# HTTP/1.1 200 OK

curl -H "Authorization: Bearer $SA_TOKEN" -H 'Host: api.toystore.com' -X POST http://localhost:9080/admin/toy -i
# HTTP/1.1 403 Forbidden

❼ Create the `RateLimitPolicy`

kubectl apply -f -<<EOF
apiVersion: kuadrant.io/v1beta1
kind: RateLimitPolicy
metadata:
  name: toystore-rate-limit
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: toystore
  rateLimits:
    - configurations:
        - actions:
            - metadata:
                descriptor_key: "userID"
                default_value: "no-user"
                metadata_key:
                  key: "envoy.filters.http.ext_authz"
                  path:
                    - segment:
                        key: "ext_auth_data"
                    - segment:
                        key: "userID"
      limits:
        - conditions: []
          maxValue: 5
          seconds: 10
          variables:
            - userID
EOF

Note: It may take a couple minutes for the RateLimitPolicy to be applied depending on your cluster.

Try the API rate limited

Send requests as the Keycloak-authenticated user:

while :; do curl --write-out '%{http_code}' --silent --output /dev/null -H "Authorization: Bearer $ACCESS_TOKEN" -H 'Host: api.toystore.com' http://localhost:9080/toy | egrep --color "\b(429)\b|$"; sleep 1; done

Send requests as the service account:

while :; do curl --write-out '%{http_code}' --silent --output /dev/null -H "Authorization: Bearer $SA_TOKEN" -H 'Host: api.toystore.com' http://localhost:9080/toy | egrep --color "\b(429)\b|$"; sleep 1; done

Each user should be entitled to a maximum of 5 requests to the API every 10 seconds.

Note: You may need to refresh the tokens if they are expired.

Cleanup

make local-cleanup