Skip to content

API Lifecycle

In only the rarest of cases will the first version of a CRD be also its final version. Instead usually CRDs evolve over time and Kubernetes has strong, though sometimes hard to use, support for managing different versions of CRDs and their resources.

To understand how CRDs work in the context of the Sync Agent, it's important to first get familiar with the regular Kubernetes behaviour regarding CRD versioning.

Basics

The Sync Agent will, whenever a published CRD changes (this can also happen when the projection rules inside a PublishedResource are updated), create a new APIResourceSchema (ARS) in kcp. The name and version of this ARS are based on a hash of the projected CRD. Undoing a change would make the agent re-use the previously created ARS (ARS are immutable).

After every reconciliation, the list of latest resource schemas in the configured APIExport is updated. For this the agent will find all ARS that belong to it (based on an ownership label) and then merge them into the APIExport. Resource schemas for unknown group/resource combinations are left untouched, so admins are free to add additional resource schemas to an APIExport.

This means that every change to a CRD on the service cluster is applied practically immediately in each workspace that consumes the APIExport. Administrators are wise to act carefully when working with their CRDs on their service cluster. Sometimes it can make sense to turn-off the agent before testing new CRDs, even though this will temporarily suspend the synchronization.

Single-Version CRDs

A very common scenario is to only ever have a single version inside each CRD and keeping this version perpetually backwards-compatible. As long as all consumers are aware that certain fields might not be set yet in older objects, this scheme works out generally fine.

The agent will handle this scenario just fine by itself. Whenever a CRD is updated, it will reflect those changes back into a new APIResourceSchema and update the APIExport, making the changes immediately available to all consumers. Since the agent itself doesn't much care for the contents of objects, it itself is not affected by any structural changes in CRDs, as long as it is able to apply them on the underlying Kubernetes cluster.

Multi-Version CRDs

Having multiple versions in a single CRD is immediately much more work, since in Kubernetes all versions of a CRD must be losslessly convertible to every other version. Without CEL expressions or a dedicated conversion webhook this is practically impossible to achieve.

At the moment kcp does not support CEL-based conversions, and there is no support for configuring a conversion webhook inside the Sync Agent either. This is because such a webhook would need to run very close to the kcp shards and it's simply out of scope for such a component to be described and deployed by the Sync Agent, let alone a trust nightmare for the kcp operators who would have to run foreign webhooks on their cluster.

Since both conversion mechanisms are not usable in the current state of kcp and the Sync Agent, having multiple versions in a CRD can be difficult to manage.

Generally the Sync Agent itself does not care much about the schemas of each CRD version or the convertibility between them. The synchronization works by using unstructured clients to the storage versison of the CRD on both sides (in kcp and on the service cluster). Which version is the storage version is up to the CRD author.

When publishing multiple versions of a CRD

  • only those versions marked as served can be picked and
  • if no storage version is picked, the latest (highest) version will be chosen automatically as the storage version in kcp.