Smarter Kubernetes Scaling with KEDA and Custom Metrics

Kubernetes adds pods when CPU or memory runs hot. A lot of background work never runs hot: a crawler waiting on the Google Drive API or a worker waiting on a database barely uses its processor, even with hundreds of tasks queued behind it. We scale that kind of service on the work that is waiting instead, with KEDA and a small metrics server of our own. More than two dozen of our services run this way, from connectors that scale to zero when idle to crawlers that burst to 50 pods.

Part one: the plain-language version. Three short sections on what goes wrong with the default and what KEDA changes. Every technical term is underlined where it first appears; hover over it, or tap it on a phone, for a definition.

Why scaling on CPU misses waiting work

Applications on KubernetesKubernetesOpen-source software that runs applications in containers across a group of servers, restarts them when they fail and adds copies when asked. run as podsPodThe smallest unit Kubernetes runs: one copy of an application, in one or more containers.: identical copies of the same program. When more work arrives you want more copies, and when it dies down you want fewer. That is autoscalingAutoscalingAdding or removing copies of an application automatically as the amount of work changes., and the autoscaler built into Kubernetes is the Horizontal Pod AutoscalerHorizontal Pod Autoscaler (HPA)Kubernetes' built-in autoscaler. It adds or removes pods to keep a metric, usually CPU or memory use, near a target.. It watches how busy each pod's processor and memoryCPU and memoryThe processor time and working memory a pod uses. These are the default autoscaler's signals. are and adds pods when they cross a target.

That works when the work is computing. Much background work isn't. A crawler copying files from Google Drive spends most of its time waiting on Google's servers. A worker that writes to a database waits on the database. Their processors stay nearly idle even with hundreds of tasks waiting, so the autoscaler sees nothing to do (Figure 1).

The useful number is the one the default autoscaler can't see: how much work is waiting. That can be the length of a queueQueueA list of tasks waiting to be picked up. How long it is makes one of the most useful scaling signals., a count of pending rows in a table, or any other metricMetricA number that describes a workload at a moment, such as CPU use or the count of tasks waiting. your own code can report.

What KEDA adds

KEDAKEDAKubernetes Event-driven Autoscaling: an open-source add-on that scales workloads on events and custom metrics, down to zero pods. is a small open-source add-on for Kubernetes that scales on that kind of number. It checks a source on a schedule, every 30 seconds by default, and passes the number to Kubernetes, whose autoscaler sets the number of pods. A rule called a ScaledObjectScaledObjectThe KEDA resource that ties a deployment to its triggers and to the fewest and most pods it may run. says what to check and how far to scale. When nothing is waiting, KEDA can scale to zeroScale to zeroRunning no pods at all while there is no work, and starting one when work arrives.: it takes a service down to no pods at all and starts one again when work arrives (Figure 2).

The source can be one KEDA already knows, such as an Amazon SQS queue or a PostgreSQL query, or a custom metrics serverCustom metrics serverA small web service of your own that answers one question for KEDA: how much work is waiting. of your own that returns a single number. Part two explains why we chose the second.

How we use it at Needl.ai

More than two dozen of our background services scale this way. All but one ask our own metrics server, a small Python service, how many tasks are waiting for them; the other uses one of KEDA's built-in scalers. Connectors for apps that are rarely busy scale to zero, and the crawlers that pull in data rooms from Google Drive, OneDrive, SharePoint, Box and Dropbox add pods for the backlog, up to 50 (Figure 3).

What this means for your team

• Data rooms sync sooner. When a deal team connects a data room on Google Drive, OneDrive, SharePoint, Box or Dropbox, the crawlers that pull it in add pods for the backlog, up to 50, and drop them once it clears.
• Reports start when you ask. Each queued report gets its own worker, up to 15 at once, so a long report does not hold up the next one.
• Quiet connectors cost nothing to keep. Connectors for apps that are rarely busy scale to zero and start again when there is work.

Ten terms to know

Part two builds on these ten. The other terms it uses are defined in the glossary at the end.

Part two: the engineering detail. Sai's original write-up: where CPU-based scaling falls short, how KEDA and a custom metrics server work together, a working ScaledObject and what it saves.

Where CPU-based scaling falls short

Running applications on Kubernetes often means dealing with tasks triggered by events or batch jobsBatch jobWork that runs through a set of items and finishes, rather than answering requests as they come., like processing queues, scheduled workloads, or interacting frequently with databases and APIs. Traditional Kubernetes scaling, using the Horizontal Pod Autoscaler (HPA), primarily relies on CPU and memory usage metrics. While this works great for compute-intensive workloads, it doesn't align well with scenarios where your applications spend most of their time waiting on external responses or handling event-drivenEvent-drivenWork that starts because something happened, such as a message landing on a queue, rather than on a timer. tasks.

In other words, your pods might stay idle or underused even as queues grow or external tasks pile up, forcing you to manually manage scaling during sudden spikes or increased workloads. You end up either provisioning more resources than necessary to handle unexpected spikes or suffering from poor responsiveness during peak loads.

This post dives into how KEDA (Kubernetes Event-driven Autoscaling) paired with custom metricsCustom metricA metric your own code computes and reports, rather than one Kubernetes measures for you. can help you scale your Kubernetes workloads more accurately, efficiently, and cost-effectively based on actual workload demands.

So, what is KEDA?

KEDA is a lightweight component built specifically for Kubernetes to enhance its autoscaling capabilities. Unlike traditional HPA, KEDA scales your applications based on real-time events and custom-defined metrics, giving you the flexibility to scale precisely according to your workload needs.

KEDA has two main components:

KEDA Controller

• Handles scaling your Kubernetes deployments up and down, even to zero pods if no work is present.
• Integrates smoothly with the existing Kubernetes Horizontal Pod Autoscaler (HPA).
• Detects events from external sources and communicates these signals directly to the HPA.

KEDA Metrics Server (Adapter)

• Collects rich, real-time metrics from external sources and converts them into Kubernetes-compatible metrics.
• Allows HPA to consume these metrics easily and make informed scaling decisions.
• Seamlessly connects with external event sources like message queues, databases, and more.

Together, these components empower your Kubernetes clustersClusterThe group of servers that Kubernetes manages as one. to scale dynamically and precisely based on the exact needs of your workloads.

Exploring KEDA scalers and custom metrics

KEDA comes with several built-in scalersScalerA KEDA plug-in that reads one kind of source, such as an SQS queue, a PostgreSQL query or a metrics API. for popular platforms such as AWS SQS, PostgreSQL, Elasticsearch, and more. While these built-in options simplify scaling, there are scenarios where having a custom metrics scaler becomes a huge advantage:

• Better security and credential managementSome built-in scalers, like the PostgreSQL scaler, require direct access to database credentialsCredentialsThe usernames, passwords or keys that grant access to a system.. With a custom metrics scaler, you can avoid exposing these credentials at the infrastructure level by connecting to the database securely through your application code, using tools like RDS ProxyRDS ProxyAn AWS service that sits between applications and a database, pools their connections and handles the credentials. or service accountsService accountAn identity for a program rather than a person, used to grant it access without a shared password..
• Less infrastructure complexityWith custom scalers, you avoid hard coding specific queries or scaling logic into TerraformTerraformA tool that describes cloud infrastructure in code files and creates it from them. or HelmHelmA package manager for Kubernetes. A Helm chart installs an application and its settings with one command. charts, making your infrastructure simpler and easier to maintain.
• Flexible metric queriesYour team can create tailored, sophisticated queries directly within your application logic, giving you exactly the metrics you need for precise scaling.

Putting it into practice: KEDA + custom metrics server

Consider implementing a custom metrics server, for instance using Python. This server securely interacts with your databases (through secure intermediaries like RDS Proxy) to gather essential metrics like pending tasks or queue lengths.

Here's how the scaling process looks in action:

1. You define scaling rules using a ScaledObject or ScaledJobScaledJobLike a ScaledObject, but it starts one Kubernetes Job per piece of waiting work instead of resizing a deployment..
2. The KEDA ControllerKEDA ControllerThe KEDA component that reads ScaledObjects, creates an HPA for each one and handles scaling to and from zero. picks up these rules and sets up an HPA resource accordingly.
3. The Controller activates the KEDA Metrics AdapterKEDA Metrics Server (Adapter)The KEDA component that fetches values from scalers and serves them to the HPA in the format Kubernetes expects. specifically for your workload.
4. The Metrics Adapter queries your custom metrics server (Python-based, in this case).
5. The metrics server returns real-time metrics.
6. KEDA Metrics Adapter forwards these metrics directly to Kubernetes HPA.
7. The HPA then scales your application pods up or down based on these precise metrics.

Behind the scenes: how KEDA triggers scaling

Quick setup: installing KEDA and defining a ScaledObject

Getting started with KEDA is simple. Here's how you can install it and define a scaling rule with a ScaledObject.

Install the KEDA operator via Helm

helm repo add kedacore https://kedacore.github.io/charts
helm repo update
helm install keda kedacore/keda --namespace keda --create-namespace

This installs a single KEDA operatorOperatorA program inside the cluster that installs and runs a piece of software for you. KEDA runs as one. pod in the keda namespaceNamespaceA named section of a cluster that keeps one group of resources apart from the rest., which includes both the controller and the metrics adapter as sidecar containersSidecar containerA helper container that runs in the same pod as the main one..

Example ScaledObject YAML

Below is a sample ScaledObject for an email-notification deploymentDeploymentA Kubernetes resource that keeps a set number of identical pods running. Autoscaling changes that number.. This uses a custom metrics API to scale based on a count value.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: email-notification  # Name of the ScaledObject (acts as the deployment identifier for this scaling rule)
  namespace: default
  annotations:
    scaledobject.keda.sh/transfer-hpa-ownership: "true"  # Ensures KEDA manages HPA ownership
spec:
  scaleTargetRef:
    name: email-notification  # Name of the Kubernetes Deployment to scale
  minReplicaCount: 1           # Minimum number of pods
  maxReplicaCount: 5           # Maximum number of pods
  advanced:
    horizontalPodAutoscalerConfig:
      name: email-notification  # Optional: custom HPA name
  triggers:
    - type: metrics-api
      metadata:
        targetValue: "10"     # When 'count' exceeds this, scale up
        activationTargetValue: "0"  # Don't scale from 0 until value > 0
        format: "json"        # Expected format from metrics API
        url: "http://custom-metrics-server.default.svc.cluster.local/count?service=email-notification"
        valueLocation: "count"  # JSON field name containing the count
        useCachedMetrics: "true"  # Enables caching to reduce metric server load

About the metrics server

You can implement a simple Python-based metrics server using Flask or FastAPIFlask and FastAPITwo widely used Python libraries for building small web services.. This server just needs to expose an endpointEndpointOne address a web service answers on, such as /count. that returns a JSONJSONA plain-text format for structured data, such as {"count": 27}. response like:

{ "count": 27 }

KEDA will regularly poll this endpoint to determine whether to scale your pods.

Real-world benefits and outcomes

By implementing KEDA with custom metrics, you'll notice several immediate improvements:

• Dynamic, real-time scalingYour pods scale up and down automatically based on the actual workload, whether it's a sudden surge in events or a spike in queued jobs. No more waking up to manually tweak pod counts or over-provisioningOver-provisioningRunning more capacity than the work needs, in case a spike arrives. "just in case." KEDA ensures the right number of pods are always running, exactly when they're needed.
• Significant cost savingsImagine a service previously running 5 pods continuously, each using 1 CPU and 2 GB of memory (totaling 5 CPUs and 10 GB memory). After introducing KEDA and HPA, this service typically scales between 1 and 5 pods based on real demand. On average, it might run just 2 pods, meaning resource consumption drops by approximately 60%, directly reducing infrastructure costs.

• Operational simplicity and flexibilityDevelopers can easily adjust scaling logic and metrics directly within the application code, without complicated infrastructure updates.
• Cloud-agnostic scalabilityBecause custom metrics avoid tight coupling with specific cloud services, your setup remains portable across different cloud environments or on-premisesOn-premisesRunning on servers an organisation owns, in its own data centre. deployments.

Wrapping up

KEDA's ability to leverage custom metrics for scaling gives your Kubernetes applications the responsiveness, security, and efficiency they need, especially when traditional CPU or memory-based scaling falls short. For anyone running event-driven or batch-processing workloads in Kubernetes, adopting KEDA with custom metrics can deliver substantial improvements in resource management, cost efficiency, and operational flexibility.

Want to learn more?

• KEDA Official Documentation
• Kubernetes HPA Documentation

Got questions?

If you have thoughts, questions, or want to share how you're using KEDA in your own stack, don't hesitate to reach out! Connect with me on LinkedIn.

FAQs

Does KEDA replace the Horizontal Pod Autoscaler?

No. KEDA creates an HPA for each ScaledObject and feeds it metrics, and the HPA still decides how many pods to run between one and the maximum. KEDA itself handles the step from zero pods to one and back, which the HPA can't do.

How quickly does KEDA react to new work?

It checks each source every 30 seconds by default, set by the polling intervalPolling intervalHow often KEDA checks a trigger's source. The default is 30 seconds., and a new pod then needs time to start. When the work stops, KEDA waits for the cooldown periodCooldown periodHow long KEDA waits after the last activity before it scales a workload to zero. The default is 300 seconds., 300 seconds by default, before scaling a service to zero, and Kubernetes waits out short dips before it removes pods.

What happens if the metrics server is down?

KEDA can't read the number, so it can't ask for more pods, and the service keeps the pods it has. KEDA's optional fallback setting can hold a fixed number of pods instead. We run two copies of our metrics server so one can restart while the other answers.

Why not use KEDA's built-in PostgreSQL scaler?

It works, but it needs database credentials where KEDA can read them, and the query written into infrastructure code. A metrics server keeps both in the application, so changing what a service scales on is an ordinary code change (Figure 4).

When should a service not scale to zero?

When its first task can't wait for a pod to start. Scaling from zero means the first task waits while a pod is scheduled and started, so we keep one pod running for services that answer requests as they arrive, and let the rarely busy ones drop to zero.

Does this only work on AWS?

No. KEDA runs on any Kubernetes cluster, and a custom metrics server is ordinary application code. In our setup, only the database access through RDS Proxy is specific to AWS.

Glossary