Performance problems rarely appear as one clear failure.

A slow database call may first show up as rising HTTP latency. Then application threads start filling up. Pending database connections increase. Memory usage climbs. Eventually, Kubernetes restarts the container.

By the time an engineer opens the dashboard, the system may look normal again.

I have spent hours comparing logs, traces, thread pools, database connections, Kafka lag, and memory graphs just to understand what happened during a short incident.

That made me ask a simple question:

Why can’t an AI agent perform the first round of this investigation?

The metrics already exist in tools like Prometheus. Experienced engineers already use repeatable diagnostic patterns. The missing piece is an agent that can connect those signals and explain what they mean.

Observe Before Recommending

A performance agent should not begin by suggesting more memory, larger pools, or additional replicas.

Its first question should be:

Do I have enough information to understand the problem?

For a Java application, that may require signals from several layers:

  • HTTP latency and throughput
  • Tomcat thread utilization
  • Database connection pool usage
  • Kafka consumer lag
  • Circuit breaker states
  • JVM and container memory
  • Downstream response sizes

If important signals are missing, the agent should stop and recommend better instrumentation.

The correct workflow is:

Instrument → Reproduce → Diagnose → Tune → Verify → Alert

Skipping directly to tuning usually creates more confusion.

Turn Metrics Into Signals

Raw metric values are not enough.

A thread count of 150 means little unless the agent also knows the maximum thread count, the previous value, request volume, and latency trend.

The agent should convert raw values into signals:

thread_utilization = busy_threads / maximum_threads

connection_utilization = active_connections / maximum_connections

It should also understand whether a metric is increasing, stable, or recovering.

Useful signals include:

  • Thread utilization above 80 percent
  • Pending connections increasing
  • Kafka lag growing
  • P99 latency rising faster than median latency
  • Memory climbing steadily
  • Circuit breakers opening
  • Containers restarting without recorded application errors

The agent should compare signals over five-minute, fifteen-minute, and thirty-minute windows.

A snapshot shows what is happening now.

A time series shows how the failure developed.

Use Little’s Law Before Increasing Pools

One of the most useful pool-sizing formulas is based on Little’s Law:

required_pool_size = requests_per_second × response_time_in_seconds

For high-throughput systems, measured P99 latency is often more useful than average latency.

Suppose an application processes 500 requests per second and its P99 response time is 10 milliseconds:

500 × 0.010 = 5

The estimated requirement is five concurrent connections. A safety buffer might increase that number to ten.

The problem begins when engineers increase pool sizes without considering every replica.

Imagine twenty Kubernetes pods with fifty database connections each:

20 × 50 = 1,000 connections

If the database supports only one hundred connections, the application is overwhelming its own dependency.

Before recommending a pool increase, the agent must always calculate:

pool_size × replica_count

It should then compare that result with the downstream system’s limit.

One Failure Can Appear in Three Places

A slow downstream dependency often creates the same pattern across multiple layers.

First, downstream calls become slower.

Application threads remain busy while waiting.

Connection pools start filling up.

Requests continue arriving faster than they can be processed.

Queues grow.

When the downstream system finally responds, many large responses may arrive together. Memory spikes. Garbage collection falls behind. The container runs out of memory and restarts.

This may look like three different problems:

  • Thread pool saturation
  • Connection pool saturation
  • Memory pressure

In reality, they may be three faces of the same failure.

The common mechanism is accumulation.

The agent should search for signals that move together:

IF thread utilization increases AND P99 latency increases AND pending connections increase THEN investigate a slow downstream dependency

Another useful rule is:

IF memory rises after a latency spike THEN inspect queued work and response sizes

This is where the agent becomes more useful than another dashboard.

A dashboard displays metrics.

The agent explains the relationship between them.

Check Payload Size Before Tuning the JVM

Performance tuning often starts with heap settings, garbage collection flags, thread counts, and connection pools.

Sometimes the real issue is much simpler.

The application is moving too much data.

Imagine a downstream API returning an 8 MB response while the application uses only 20 percent of the fields.

With one hundred concurrent requests, the system may allocate hundreds of megabytes for data that will be discarded after deserialization.

Increasing the heap may delay the failure, but it does not solve the cause.

The agent should ask:

  • How large are the responses?
  • Did response size increase before memory usage increased?
  • How much of the payload is actually used?
  • Can the API return only required fields?

Shrinking payloads can improve network usage, deserialization time, memory pressure, garbage collection, and latency at the same time.

Protect Downstream Calls in Layers

Timeouts alone are not enough.

If a service becomes slow, hundreds of threads may remain blocked until their timeouts expire.

Circuit breakers alone are also not enough. The application may consume many threads before the breaker opens.

A resilient system usually needs three layers:

Timeouts limit how long one call can hold a thread.

Circuit breakers stop repeated calls to a failing dependency.

Resource isolation prevents one dependency from consuming every available thread.

Each layer handles a different failure mode.

The agent should check all three before recommending more capacity.

Build the First Agent With Rules

The first version does not need a complex machine learning model.

A rule-based system can already provide useful diagnoses.

First, retrieve aligned time-series metrics from Prometheus through MCP or another integration.

Next, convert raw values into ratios, trends, and states.

Then apply diagnostic rules:

IF threads rise AND latency rises AND pending connections rise THEN downstream bottleneck causing accumulation

IF Kafka lag rises AND processing time rises AND CPU remains stable THEN investigate blocking I/O or a slow dependency

The agent must correlate web, database, messaging, resilience, and infrastructure signals.

It should then explain the diagnosis in plain language:

Thread utilization reached 92 percent while pending database connections increased from zero to fourteen. During the same period, P99 latency increased from 45 milliseconds to 380 milliseconds. This pattern suggests requests are waiting on a downstream database operation. Check slow queries and database health before increasing the connection pool.

That is far more useful than dumping another collection of charts.

Add Guardrails Before Recommendations

Before suggesting any change, the agent should validate its assumptions.

It should check:

  • Total connections across replicas
  • Downstream capacity
  • Payload sizes
  • Missing metrics
  • Actual throughput
  • P99 latency
  • Memory headroom

The agent should also be allowed to say:

There is not enough evidence to recommend a change.

A cautious answer is better than a confident recommendation that creates another outage.

Final Thoughts

Performance troubleshooting is mostly pattern recognition.

Engineers compare timelines, connect signals, form a hypothesis, and test it.

A performance analysis agent can automate the first part of that process.

Start with reliable instrumentation. Convert metrics into trends. Apply simple engineering rules. Correlate evidence across layers. Add guardrails before making recommendations.

The goal is not to build another dashboard.

The goal is to build a system that understands what the dashboard is trying to say.