Kyverno Generate, Part 2: Label-Driven Self-Service Resources

28.09.2020 — kubernetes, kyverno, policy, self-service — 5 min read

Part 1 covered the basics of Kyverno generate rules: creating and synchronizing resources across namespaces. This part covers a more powerful pattern: triggering generation from a label, so a namespace owner opts into a capability by adding a label rather than filing a ticket. It is the foundation of self-service platform features like "add this label to get backups."

Examples target the current Kyverno v1 policy API (kyverno.io/v1).

TL;DR

A generate rule can match on a label selector, not just a resource kind, so it fires only when a namespace carries a specific label.
This turns generation into self-service: the owner adds backup=enabled and gets a backup schedule, removes it and the schedule goes away (with synchronize: true).
Generation is done by the kyverno-background-controller ServiceAccount. Grant it permission to create the target type with an aggregation-labeled ClusterRole.
Use the current match.any syntax with a selector under resources, not the older bare match.resources.

The problem: opt-in without tickets

A platform team wants namespace owners to be able to turn features on and off themselves: backups, monitoring, a service account for scraping. The team does not want to hand-create those resources, and it does not want a ticket queue. A label is the natural switch: the owner sets it, and the platform reacts.

Kyverno generate rules with a label selector make the label the trigger. Add the label, the resource appears. Remove it, and if the rule synchronizes, the resource is removed.

How a label-triggered rule works

The match block selects namespaces by label. When a namespace is created or updated so that it matches the selector, the rule fires. When it stops matching (the label is removed), synchronization deletes the generated resource.

owner adds label backup=enabled to a namespace
        │  namespace matches match.any selector
        ▼
  Kyverno generates the backup Schedule
        │
   owner removes the label
        ▼
  namespace no longer matches ──▶ Schedule deleted (synchronize: true)

Caption: the label is the switch; synchronization removes the resource when the label goes away.

Walkthrough: self-service backups with Velero

Velero backs up Kubernetes resources and volumes. A namespace owner opts into a daily backup by adding backup.example.com/enabled: "true". The rule generates a Velero Schedule in the velero namespace scoped to their namespace. Note the current match.any with a selector under resources.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: add-velero-autobackup
spec:
  rules:
    - name: gen-velero-schedule
      match:
        any:
          - resources:
              kinds:
                - Namespace
              selector:
                matchLabels:
                  backup.example.com/enabled: "true"
      generate:
        apiVersion: velero.io/v1
        kind: Schedule
        name: "{{request.object.metadata.name}}-auto-schedule"
        namespace: velero
        synchronize: true
        data:
          spec:
            schedule: "0 0 * * *"
            template:
              includedNamespaces:
                - "{{request.object.metadata.name}}"
              snapshotVolumes: true
              ttl: 168h0m0s

Notice the selector.matchLabels under resources is what makes this label-driven. With synchronize: true, removing the label removes the Schedule, so opt-out is as easy as opt-in.

Walkthrough: a Prometheus ServiceAccount on demand

The same pattern provisions a ServiceAccount when a namespace opts into monitoring with monitoring.example.com/enabled: "true".

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: add-monitoring-sa
spec:
  rules:
    - name: gen-prometheus-sa
      match:
        any:
          - resources:
              kinds:
                - Namespace
              selector:
                matchLabels:
                  monitoring.example.com/enabled: "true"
      generate:
        apiVersion: v1
        kind: ServiceAccount
        name: prometheus-scraper
        namespace: "{{request.object.metadata.name}}"
        synchronize: true
        data:
          metadata:
            labels:
              app.kubernetes.io/name: prometheus

Notice the trigger and target namespaces are the same here (request.object.metadata.name), unlike the Velero example where the Schedule lives centrally in velero.

RBAC: let the background controller create these types

Both examples generate resource types the background controller does not manage by default: Velero Schedule (a custom resource) and ServiceAccount. Grant the permissions with an aggregation-labeled ClusterRole, which Kyverno merges into the background controller's role automatically (customizing permissions).

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kyverno:generate-backup-monitoring
  labels:
    rbac.kyverno.io/aggregate-to-background-controller: "true"
rules:
  - apiGroups: ["velero.io"]
    resources: ["schedules"]
    verbs: ["create", "update", "delete", "get", "list", "watch"]
  - apiGroups: [""]
    resources: ["serviceaccounts"]
    verbs: ["create", "update", "delete", "get", "list", "watch"]

Notice the label rbac.kyverno.io/aggregate-to-background-controller: "true". Add a new role like this rather than editing Kyverno's built-in roles, so upgrades do not overwrite it. Also create the velero namespace first, since the Schedule is generated there.

Failure modes and gotchas

The rule does not fire on an existing namespace. A label-selector rule reacts to the label being added. For namespaces that already carry the label when you install the policy, set generateExisting: true so it applies retroactively (see part 1).
The resource is not created. Check kubectl get updaterequests -A; a Failed status almost always means the background controller lacks permission on the target type. Add the aggregation-labeled ClusterRole.
Removing the label does not clean up. That only happens with synchronize: true. Without it, the generated resource is left behind.
Old syntax from 2020 tutorials. Bare match.resources with a selector, and the kyverno:generatecontroller ClusterRole, are both outdated. Use match.any and the aggregation label shown here.
validationFailureAction on a generate policy. That field is for validate rules and does nothing for a generate-only policy; leave it off.

Summary and next steps

Label-driven generate rules turn Kyverno into a self-service platform layer: a namespace owner adds a label and gets backups, monitoring, or any other resource, and removes the label to opt out. The mechanics are the same as part 1, with a selector on the trigger and the background controller granted permission to create the target type. Combine this with generateExisting: true to apply it to namespaces that already exist.