Scaling Limits of the Kubernetes Resource Model & Operator Pattern

The Control Plane Scaling Stack

The Kubernetes control plane has no single "max capacity" setting. Instead, practical limits emerge from a stack of intersecting constraints — etcd storage at the bottom, API server concurrency in the middle, and client-side rate limiting at the top. Understanding which layer is the bottleneck for your workload is the first step toward scaling.

Volume Limits — Hard Ceilings

The practical ceiling on how many Custom Resources, CRDs, and controllers a cluster can host is determined by etcd storage, API server memory, and informer cache overhead — not by any single configurable max.

CRs per Resource Type — The Storage Math

GKE enforces a per-resource-type budget of 800 MB in etcd. The number of Custom Resources you can store depends entirely on object size:

CRD Count Degradation Curve

Crossplane's testing revealed a non-linear degradation pattern as CRDs increase. The problems appear in distinct phases:

Three Layers of Rate Limiting

Operator throughput is governed by three independent, multiplicative rate-limiting layers. Tuning only one while ignoring the others produces no visible improvement — you must identify which layer is actually the bottleneck.

Layer 1 — client-go REST Rate Limiter

The tightest constraint for most operators. The default token bucket allows 5 requests per second with a burst of 10. A controller making 3 API calls per reconciliation (get, patch status, create child) maxes out at roughly 1.5 reconciliations per second.

Layer 2 — controller-runtime Workqueue Rate Limiter

A MaxOfRateLimiter combining per-item exponential backoff (base 5 ms, max 1,000 s) with a global token bucket (10 QPS, burst 100). Prevents a failing CR from thrashing while capping thundering herd on restart.

Layer 3 — API Priority & Fairness (APF)

Server-side protection distributing 600 total inflight requests (400 non-mutating + 200 mutating) across priority levels. GA since Kubernetes 1.29. When exceeded, clients receive HTTP 429.

Key Metrics & KPIs

The right metrics provide early warning of scaling issues before outages. They fall into three domains: controller health, API server pressure, and etcd capacity.

Controller Health — Leading Indicators

etcd Capacity — The Most Consequential Alerts

API Server Pressure — SIG Scalability SLOs

Published Benchmarks & Research

The most rigorous scaling data converges on a consistent set of boundaries. Here are the key findings from each source.

SIG Scalability — Official Envelope

Benchmark Timeline

Upstream Fixes That Raised the CRD Ceiling

Failure Cascades

Each failure mode follows a characteristic cascade pattern. Understanding the sequence is more valuable than knowing the threshold in isolation.

The Thundering Herd Cascade

The most common catastrophic failure. Triggered by API server restart, network partition recovery, or etcd compaction.

Informer Cache OOM Anti-Pattern

Reconciliation Drift Under Load

Failure Mode Summary

Scaling Strategies — Ranked by Impact

The most impactful scaling techniques are architectural decisions made early, not parameter tuning applied later. Evidence from production operators points to a clear hierarchy.

1. Narrow Informer Scope

The single change with the largest impact. Configure cache.Options with per-GVK label selectors, field selectors, and namespace restrictions. Use SetTransform to strip managed fields, annotations, and last-applied-configuration before caching. Routinely reduces memory by 80–90%.

2. Server-Side Apply (SSA)

SSA's critical scaling property: no-op writes are not stored in etcd or broadcast to watchers. This eliminates the hot-loop anti-pattern where a controller writes back an unchanged object, triggering its own watch. Also removes the last-applied-configuration annotation (which can grow very large).

3. Controller Sharding

For >10,000 CRs. The most battle-tested approach is label-based sharding: a sharder assigns objects to instances via a shard label, and each instance uses a label selector to watch only its subset.

4. Rate Limiter Tuning

Start with client-go REST limits — increasing from 5/10 to 50/100 QPS/burst is safe for most clusters and immediately unlocks 10× more throughput. Then increase MaxConcurrentReconciles to 5–10 (same object is never reconciled concurrently due to workqueue dedup).

5. Predicate Filters

predicate.GenerationChangedPredicate skips status-only updates (which don't change .metadata.generation). Custom predicates can filter annotation-only changes and other noisy updates. Predicates don't reduce cache size — they prevent unnecessary reconcile enqueues.

The Scaling Landscape

The Kubernetes operator scaling landscape is defined by intersecting constraints rather than any single hard limit. etcd's 8 GB ceiling and its inability to scale horizontally remain the fundamental bottleneck — Google's solution at 65,000+ nodes was to replace etcd with Spanner entirely.

Below that threshold, the client-go 5 QPS default is the most commonly encountered bottleneck, silently throttling operators that could otherwise handle 10–50× more throughput. The informer-caches-everything design, combined with controller-runtime's implicit informer creation on any Get() call, is the most frequent source of OOM kills in production operators.

The most valuable insight from production experience: architectural decisions dominate parameter tuning. Narrowing informer scope, adopting Server-Side Apply, and implementing controller sharding each deliver order-of-magnitude improvements. Rate limiter tuning provides 5–10× gains. Together, these push a well-designed operator from thousands to hundreds of thousands of CRs — but reaching millions requires stepping outside the standard KRM model entirely.