Skip to content

ADR-040: Admission Webhooks for Resource Validation

Category: architecture Provenance: human

Decision

Implement Kubernetes admission webhooks to validate Custom Resource specifications before they are persisted to etcd. Use cert-manager for TLS certificate management instead of Kopf's auto-managed webhooks (see ADR-065 for details).

Rationale

Pydantic validation happens during reconciliation, which means: 1. Users see "resource created successfully" but it fails later 2. No immediate feedback on typos or invalid values 3. Poor user experience debugging failed resources

Admission webhooks solve this by validating at admission time: 1. kubectl create fails immediately with clear error message 2. Users get instant feedback on what's wrong 3. Invalid resources never enter etcd 4. Better GitOps experience (ArgoCD shows validation errors immediately)

Additionally, webhooks enable enforcement of: - Resource quotas (e.g., max realms per namespace) - Cross-resource constraints (e.g., one Keycloak instance per namespace) - Namespace-based authorization (before resource is created)

Using cert-manager for certificates (instead of Kopf auto-management): - Kopf's auto-management depends on insights.ready_resources which doesn't complete - cert-manager is a standard Kubernetes pattern - Allows manual control over ValidatingWebhookConfiguration - See ADR-065 for full technical rationale

Update (2025-11-12): Changed from Kopf auto-management to cert-manager based on ADR-065 findings that Kopf's auto-management doesn't work in our setup.

Consequences: - Immediate validation feedback (better UX) - Resource quotas prevent namespace abuse - One-Keycloak-per-namespace prevents conflicts - GitOps tools show validation errors immediately - Invalid resources never stored in etcd - Reliable certificate rotation via cert-manager - Requires cert-manager as cluster dependency - Increased complexity (webhook server, RBAC, certificates, cert-manager) - Bootstrap coordination needed (operator must be ready before CRs) - Admission webhook failures block resource creation (fail-closed by default) - Can disable webhooks if cert-manager not available (fallback to reconciliation validation)

Related Decisions: - ADR-033: Prohibits webhooks for configuration API. This ADR adds webhooks for validation only. - ADR-065: Specifies cert-manager for webhook certificates instead of Kopf auto-management.

Agent Instructions

When implementing resource validation, use both Pydantic models (for type safety in code) AND admission webhooks (for early validation at K8s API boundary). Webhooks should validate specs synchronously and return clear error messages. Use kopf.on.validate decorators with explicit 'id' parameter for webhook paths. Ensure cert-manager is available for TLS certificate generation. Include readiness probe that checks webhook server port.

Implementation Details: - Bootstrap consideration: The Keycloak CR in the operator Helm chart must be created AFTER the operator pod is ready and webhook server is listening. This is handled by: 1. Enhanced readiness probe checking webhook port 8443 2. Helm --wait flag ensures operator ready before Keycloak CR 3. ArgoCD sync waves (operator=wave0, Keycloak=wave1) 4. Webhook readiness check in test fixtures (conftest.py)

  • RBAC requirements:
  • create/update/delete for validatingwebhookconfigurations

  • create/update/delete for mutatingwebhookconfigurations (for future use)

  • Dependencies:

  • cert-manager 1.14+ (for TLS certificate generation and rotation)
  • Certificates mounted at /tmp/k8s-webhook-server/serving-certs

Rejected Alternatives

Pydantic validation only

No immediate feedback to users. Resources appear created but fail during reconciliation. Poor UX for debugging.

Use Kopf auto-managed webhooks with self-signed certs

Kopf's auto-management depends on insights.ready_resources.wait() which never completes in our operator setup. See ADR-065 for details.

Disable webhooks, only use CRD OpenAPI schema validation

CRD validation is limited (no cross-resource checks, no quotas, no complex logic). Webhooks enable richer validation.

Manual certificate management without cert-manager

Requires custom rotation logic. cert-manager is a standard solution.