PKI Storm: Managing 100,000 Simultaneous Certificate Requests in Kubernetes Recovery Scenarios

In large enterprises, TLS certificates are required everywhere: for services, workstations, containers, internal APIs, and system integrations. When thousands of consumers exist, PKI must handle massive request flows rather than individual certificates. One organization processed approximately 320 million certificate issuance requests per year.

Automation relied on the ESAUS central management system, which validates requests against security policies and forwards them to external certificate authorities without issuing certificates itself. For Kubernetes with Istio, the Citadel component of ESAUS was configured as the certificate source instead of Istio’s built-in CA. Citadel validates requests and context before passing them through ESAUS to corporate CAs, integrating Kubernetes into the enterprise PKI.

This created a dependency: Kubernetes recovery speed became tied to external CA performance. After a data center outage, dozens of thousands of pods restarted simultaneously. Because services used mTLS, each pod needed its own certificate and private key. In normal operation requests were spread over time, but during recovery roughly 100,000 requests arrived in a short window.

At 70 RPS the theoretical processing time reached almost 24 minutes. Timeouts triggered retries, amplifying load in a thundering herd effect. The infrastructure recovered faster than the CA could issue certificates, revealing that average load figures (10–11 RPS) did not predict burst behavior.

Vertical or horizontal CA scaling was evaluated. Reaching 1,050 RPS would theoretically cut storm time to 1.5 minutes, yet such capacity would remain idle more than 99 % of the time. The team instead decoupled issuance from consumption by pre-generating certificates and keys during normal operation and storing them in a secure reserve.

The reserve resides in the new Unified Secret Storage (ЕХС) layer, which provides centralized protected storage, strict authentication, access controls, auditing, high read throughput, and object lifecycle management. In performance mode, ЕХС delivers up to 14,000 RPS for existing secrets from standby nodes, allowing the burst to be served from reads rather than real-time CA operations.

The approach also addressed Key Recovery Agent (KRA) protection by keeping the KRA private key outside the CA and releasing it only for approved recovery operations. Over time the same storage layer absorbed additional operational secrets beyond the emergency reserve, evolving into a broader corporate secret management platform.

Key lessons include recognizing that average load poorly predicts disaster behavior, that some bottlenecks are better solved by separating production and consumption than by scaling, and that every architectural mitigation transfers complexity rather than eliminating it.