Skip to content

System Workspaces

System workspaces are special logical clusters that exist on every shard (including the root shard). They are used internally by kcp to store system-level resources and are not part of the regular workspace hierarchy.

Key Characteristics

All system workspaces share the following properties:

  • Naming convention: All system workspace names are prefixed with system:.
  • Shard-local: Each shard has its own independent copy of every system workspace.
  • Not user-accessible: System workspaces are only accessible to shard-local admin users (e.g. members of system:masters). There is no Workspace object representing them and no validation that a given system workspace actually exists.
  • Cannot be migrated: System workspaces are permanently bound to their shard and are never subject to workspace migration or inactivation.
  • Excluded from replication: Objects stored in system workspaces are not replicated to the cache server.
  • No workspace content authorization: The standard workspace content authorizer does not apply. Access requires global privilege (e.g. shard admin) that bypasses authorization entirely.

System Workspaces Overview

The following system workspaces exist:

Workspace Purpose
system:admin Shard-scoped administrative resources (e.g. leader election leases)
system:system-crds Central repository for all CRDs owned and managed by kcp
system:bound-crds Storage for CRDs dynamically created from APIBindings
system:shard Holds API bindings for root APIs required by every shard
system:cached-crds Used by the cache server to manage CRDs for cached resources

system:admin

The system:admin workspace holds administrative objects scoped to the local shard. It is accessible via /clusters/system:admin.

Its primary use is leader election: when kcp is deployed with multiple instances sharing the same backing storage, controllers use Lease objects in system:admin to coordinate which instance is the active leader. The relevant CLI flags are:

  • --enable-leader-election – enables leader election for kcp controllers running in the system:admin workspace.
  • --leader-election-namespace – the namespace within system:admin to use for the leader election Lease.
  • --leader-election-name – the name of the Lease object.

The system:admin context in the generated admin kubeconfig uses a separate shard admin identity, distinct from the kcp admin identity used for the root context.

system:system-crds

The system:system-crds workspace is the central store for all CRDs that kcp itself owns and manages. These CRDs are bootstrapped during kcp startup and should never be installed in any other logical cluster.

This is where kcp's own APIs live – including apis.kcp.io, core.kcp.io and cache.kcp.io resources. The exact list of CRDs is defined in config/system-crds/bootstrap.go.

CRD Resolution Priority

When kcp resolves which CRDs are available in a given workspace, it uses a three-tier priority system:

  1. System CRDs (highest priority) – CRDs from system:system-crds are always included and override everything else.
  2. APIBinding CRDs – CRDs from system:bound-crds that were created through APIBindings are added next, unless they conflict with a system CRD.
  3. Local workspace CRDs (lowest priority) – CRDs installed directly in the workspace are only included if they do not conflict with system or APIBinding CRDs.

This priority system ensures that kcp's core APIs are always available and cannot be shadowed by user-installed CRDs.

Bootstrapping

System CRDs are bootstrapped during kcp startup. The bootstrap process continuously retries until all CRDs are successfully created, blocking server startup until complete. This ensures that the shard is fully functional before it begins serving requests.

system:bound-crds

The system:bound-crds workspace stores CRDs that are dynamically created when users create APIBindings. When an APIBinding references an APIExport, the API binding controller converts the exported APIResourceSchema objects into CRDs and stores them in this workspace.

Each bound CRD is:

  • Named after the schema UID of the originating APIResourceSchema.
  • Decorated with an identity hash annotation so that the storage layer can assign the correct etcd resource prefix, keeping data from different APIExports separated.
  • Lifecycle-managed – when an APIBinding is being deleted, the bound CRD is updated with a DeletionTimestamp and a terminating condition so that the create verb is suppressed while existing data can still be read.

system:shard

The system:shard workspace holds essential API bindings that every shard needs to function. During shard bootstrap, API bindings for the following API groups are created in this workspace:

  • shards.core.kcp.io
  • tenancy.kcp.io
  • topology.kcp.io
  • cache.kcp.io
  • migration.kcp.io

These bindings ensure that every shard – including the root shard – has access to the root APIs required for workspace scheduling, shard management and cache coordination. Additionally, the shard bootstrap creates a default namespace and a LogicalCluster resource in this workspace.

system:cached-crds

The system:cached-crds workspace is specific to the cache server. It manages CRDs for resources that are replicated across shards via the cache server.

The cache server's CRD lister operates with a two-tier lookup:

  1. System CRDs from system:system-crds (same as the main kcp server).
  2. Synthetic CRDs derived from CachedResource objects. These are not real CRDs stored in etcd, but are dynamically constructed from CachedResource metadata (resource kind, scope and identity hash).

This workspace is used only by the cache server and is not relevant to regular kcp operation.

Cache Server System Resources

The cache server also bootstraps its own set of CRDs into system:system-crds (on the cache server's shard, named system:cache:server). In addition to the CRDs that the main kcp server bootstraps, the cache server's system:system-crds includes CRDs for:

  • logicalclustermigrations (migration.kcp.io)
  • shards (core.kcp.io)
  • cachedobjects (cache.kcp.io)
  • workspacetypes (tenancy.kcp.io)
  • RBAC resources (roles, clusterroles, rolebindings, clusterrolebindings)
  • Admission resources (mutatingwebhookconfigurations, validatingwebhookconfigurations, mutatingadmissionpolicies, mutatingadmissionpolicybindings, validatingadmissionpolicies, validatingadmissionpolicybindings)

These CRDs have their schemas stripped down (preserving unknown fields, no subresources) since the cache server stores replicated copies and does not need full validation.