What Telecom Monitoring Taught Me About Debugging LLM Agents

In 2021 I led an IBM team building business activity monitoring for Bell, the Canadian telecom operator. The work was deeply unfashionable: wiring Java applications into Kafka queues, designing Logstash pipelines, building Kibana dashboards, tuning alerts until they stopped crying wolf. Nobody writes breathless posts about Logstash pipeline design.

Four years later I spend my days on an agentic AI platform, and I keep having the same disorienting experience: watching very smart people rediscover, from first principles, things that operations engineering settled a decade ago. The vocabulary is new. The problems mostly are not.

Here's what transfers.

A trace is not a log, and most teams only have logs

The first thing that goes wrong in a production agent system is that something produces a bad output and nobody can explain why. The team goes looking and finds... a log line with the final response in it. Maybe the prompt. If they were careful, the model name.

That's not enough, and it was never enough. In distributed systems we learned this painfully: when a request crosses eight services, per-service logs tell you nothing, because the failure is in the composition, not in any component. You need a correlation ID threaded through the whole path and a trace you can reconstruct end to end.

An agent run is a distributed system. It just happens to run mostly in one process. A single "summarize this contract and flag the risky clauses" request might involve a retrieval step, three tool calls, a planning hop, two retries after a malformed JSON response, and a final synthesis. If you only capture the last step, you are debugging with the last frame of the film.

What you actually want to persist, per run, with a shared trace ID:

Every prompt sent and completion received — the full text, not a truncated preview
Which prompt version produced it (more on this below)
Every tool invocation with its arguments and its return value
Token counts and latency per call, not just per run
Retries and what triggered them
The model and parameters actually used, resolved at call time

Tools like LangFuse do a lot of this for you now, and they're worth adopting. But the principle matters more than the tool: if you can't reconstruct the full causal chain of a single run after the fact, you don't have observability, you have logging.

Alert on distributions, not on events

On the Bell project the most valuable thing we built wasn't a dashboard. It was the machine-learning jobs that flagged when a metric's distribution shifted — not when a single transaction failed.

This is exactly the shape of the LLM monitoring problem, and it's why naive alerting fails so badly here. A single weird LLM output is not a bug. It's the expected behaviour of a non-deterministic system. Alerting on it produces noise, the team mutes the channel, and then the real regression sails through unnoticed. I've watched this happen.

What's meaningful is a change in the shape of things:

Average output length drops 30% overnight — often the first sign a provider silently changed a model behind a stable alias
Tool-call failure rate moves from 2% to 7%
The rate at which a run hits its retry ceiling starts creeping
P95 token consumption per request climbs while request volume is flat
Refusal or empty-response rate rises on a prompt path that was stable

None of these are visible in a single trace. All of them are obvious in a time series. Instrument the aggregate, and set your alert thresholds on drift rather than on absolute values — because you rarely know the correct absolute value in advance.

Prompts are deployable artifacts. Treat them like it.

Here is a pattern I see constantly and it makes me wince every time: prompts living as string literals in application code, edited directly, shipped without review, with no record of what changed when.

Imagine proposing that for a database migration. Or a Logstash pipeline config. In the enterprise world these things are versioned, reviewed, staged, and rollback-able, because we learned that untracked configuration changes are the single most common cause of "it worked yesterday."

A prompt is configuration that materially determines system behaviour. So:

Version it, with an identifier that lands in every trace it produced
Review changes like code — a colleague should see the diff
Run your eval suite against the change before it ships
Keep the ability to roll back a prompt independently of the application

The payoff arrives the first time quality degrades and someone asks "what changed?" If prompt versions are in your traces, that's a query. If they're not, it's an archaeology project across three weeks of commits.

Evals are regression tests that admit uncertainty

The objection I hear is that you can't unit-test an LLM because the output isn't deterministic. True, and irrelevant. We've never been able to write exact-match assertions against a fraud-scoring model or a search relevance ranker either. The discipline that emerged there applies cleanly:

Build a fixed set of representative inputs with known-good characteristics. Score outputs on properties rather than exact text — did it return valid JSON against the schema, did it cite a real source document, did it decline when it should have declined, is it within a sane length band. Track the aggregate score over time. Fail the build when it drops below a threshold you've agreed on.

Fifty well-chosen cases will catch most real regressions. The most common failure mode isn't a bad eval suite — it's not having one, because the team is waiting until they can build the perfect one. Start with the twenty cases you already argue about in review, and grow the set every time production surprises you. That last part is the whole trick: every production incident should become an eval case.

Cost is a first-class reliability metric

In telecom, capacity planning was a permanent background concern. In LLM systems, cost behaves like a capacity problem with a much shorter fuse — and it's frequently the thing that kills a project after launch, not during it.

An agent with a retry loop and a large context window has an unbounded worst case. A retrieval step that quietly starts pulling twenty documents instead of five multiplies your bill without any error surfacing anywhere. Nothing alerts, nothing fails, the invoice just arrives.

So: track cost per run, not just aggregate spend. Set hard ceilings on retries and context size, and treat hitting them as an incident worth investigating rather than a routine event to swallow. Alert on cost-per-request drift the same way you'd alert on latency drift. And know your unit economics per feature — if you can't say what a single user action costs you, you can't make an informed decision about which model it should run on.

Contain the non-determinism

The broadest lesson, and the one I'd keep if I could keep only one.

LLMs introduce genuine non-determinism into software. The engineering response is not to make peace with unpredictability everywhere — it's to push the non-deterministic part into the smallest possible box and keep everything around it boring.

Queueing, retries, persistence, auditability, access control, idempotency: all of that should be conventional, testable, and thoroughly unexciting. The LLM does the judgment step. The system around it behaves exactly like any other system you'd be willing to put on-call rotation for.

This is also the clearest architectural smell I look for when reviewing an agent system. If non-determinism has leaked into the control flow — if the model is deciding whether to retry, or improvising the shape of its own output schema, or choosing whether an operation was important enough to log — you will not be able to reason about that system, and neither will anyone who inherits it.

None of this is new

The gap between an impressive demo and a system a business can depend on has always been made of the same material: observability, testing, versioning, cost control, failure containment. That was true for the Java systems I built for national infrastructure and it's true for agents.

What's genuinely new is the judgment layer, and it's remarkable. What's not new is everything required to make it dependable. If your team has people who've operated serious systems before, they know more about production AI than they think they do — the knowledge just needs translating.

I'm Tihomir Tomašević, a software architect with 17+ years in enterprise systems, currently leading development of an agentic AI platform. Through T2 Software I take on selected consulting work on exactly these problems. If you're wrestling with any of this, I'd genuinely like to hear about it — get in touch or find me on LinkedIn.