VPA vs HPA: rebalance first, then scale out

What is the VPA?

The Vertical Pod Autoscaler (VPA) is an open-source add-on from the Kubernetes autoscaler project. It sets the CPU and memory requests of a workload’s containers to match the use it measures, and keeps each container’s limit in the same ratio to its request. You attach it to a workload, such as a Deployment, with a VerticalPodAutoscaler object.

It isn’t part of core Kubernetes. On AKS it comes preconfigured in AKS Automatic, and you enable it yourself in AKS Standard. On EKS you deploy it from the autoscaler project, with Metrics Server installed.

Three components do the work:

• Recommender: reads each container’s current and past use, and its out-of-memory kills, and works out a target request.
• Updater: finds pods whose requests are out of date and evicts them, so their controller recreates them.
• Admission controller: a webhook that writes the target requests into each pod as it is created.

What happens to running pods depends on its update mode:

• Off: it only recommends. You read the targets in the VPA object.
• Initial: it sets requests when a pod is created, and never changes them later.
• Recreate: it evicts pods whose requests differ enough from the target, and they come back at the new size, within their PodDisruptionBudget.
• InPlaceOrRecreate: it changes requests without restarting the pod where the cluster supports in-place resize, and evicts the pod where it can’t. The older Auto mode now means Recreate.

The VPA does one job well: it makes requests match use. It doesn’t add pods when load grows past what one pod can serve, and it doesn’t remove nodes.

What is the HPA?

The Horizontal Pod Autoscaler (HPA) is built into Kubernetes. It is an API object, the HorizontalPodAutoscaler, and a controller that runs in the cluster’s control plane. It scales a workload out and in: it changes the replica count of a Deployment, StatefulSet or other scalable workload to hold a metric at a target. It can’t scale a DaemonSet, which runs one pod on each node.

Every 15 seconds by default, the controller reads the metric and works out how many pods the workload needs. For CPU and memory the metric is utilization: usage divided by the request, read through the Kubernetes resource metrics API, which Metrics Server usually serves. Its core rule is:

desiredReplicas = ceil(currentReplicas × currentMetric ÷ targetMetric)

With a 70% CPU target, 4 pods running at 140% of their request become 8.

The HPA can also scale on custom and external metrics, such as requests per second or queue length. KEDA builds on it: a KEDA ScaledObject creates and drives an HPA from event sources, and can scale a workload to zero.

The HPA does one job well: it matches the number of pods to load. It takes each pod’s request as given.

Every container can set a CPU and a memory request. The scheduler places pods by their requests, not by their use: a node is full when the requests on it add up to its allocatable capacity, even if the processes on it are idle. The Cluster Autoscaler, Node Auto Provisioning (NAP) and Karpenter count requests the same way when they decide how many nodes you need. A request set “to be safe” at four times real use keeps nodes running that do almost nothing, and you pay for those nodes.

Limits are different. A memory limit is the point where the container is OOM-killed, and a CPU limit is where it is throttled.

Run both on the same metric and they feed each other. Say a pod uses 200m of a 500m CPU request: 40% utilization. The VPA lowers the request to 250m. The HPA now reads the same load as 80% and, with a 70% target, adds pods. The new pods share the load, each pod uses less, and the VPA lowers the request again. The two controllers chase each other, and the replica count reflects the last request change more than real load.

That’s why the VPA’s own documentation says not to use it with the HPA on the same resource metric, CPU or memory. The usual workarounds are to run the HPA on a custom or external metric while the VPA sizes CPU and memory, or to keep the VPA in Off mode and set requests from its targets by hand.

The HPA’s arithmetic is only as good as the request under it, so the order matters.

1. Size requests to measured use: a high percentile of CPU over several days plus headroom, and the peak of memory. Leave limits alone until you know why they were set.

2. Then set the HPA target. With honest requests, 70% utilization means 70% of what a pod really needs, and the HPA scales out when load really grows.

3. Then let the node autoscaler remove what’s left. Smaller requests fit on fewer nodes, and the Cluster Autoscaler, NAP or Karpenter can remove the empty ones.

Cutting requests saves nothing on its own. The bill falls when nodes go.

Coral does the VPA’s job and works with the HPA, so you don’t have to choose between them.

• Rebalancing. Coral sets requests with its own recommender and webhook. It doesn’t run the VPA. On the default balanced policy, the CPU request is the 80th percentile of the last 3 days plus 10%, and memory is the highest use in the last 8 days plus 10%. Your manifests and Git stay as they are: a webhook sizes pods as they start, and it fails open. With automation on, Coral switches an active VPA to Off and restores it when you switch automation off or disconnect.
• HPA-aware sizing. The HPA decides how many pods run, and Coral decides how big each pod is. For a workload with a CPU-based HPA, Coral replays the last 3 days of load through that HPA’s own settings before it picks a request. A smaller request wins only if its saving beats the cost of unserved load and extra scaling. While the HPA is scaling up and below its max, Coral holds back its surge and CPU-pressure raises, and pauses pod replacements until every pod is Ready. An OOM kill still raises memory at once.
• HPAs it manages. With automation on and the Optimization type “Both”, Coral creates an HPA, with minReplicas of 2 or more, for an eligible Deployment or Argo Rollout that has none. Eligible means 24 hours of history, at least 100m CPU per pod and 2 or more ready replicas. It takes over hand-made HPAs that nothing else manages, keeps your minReplicas, only raises maxReplicas, and restores your values when it hands them back. Workload optimization never changes HPAs that Helm, Argo CD, Flux, KEDA or kubectl apply manage.
• Opt-in. Coral only recommends until “Automatically optimize” is on for a policy or a workload.