70% of engineering groups use at least four observability systems at once, according to a 2024 Grafana Labs study. Yet seeing this as progress misses the point entirely. The same study cited 62 different observability technologies in use across respondents, highlighting how fragmented monitoring stacks have become. Despite all those layers, outages still slip through unseen.
It’s not a data problem. Teams are drowning in signal-to-noise. When something breaks, engineers jump between three dashboards trying to line up timestamps manually because the tools don’t sync. As one engineering team put it: ‘We had alerts for everything, which meant we had alerts for nothing. The team had learned to ignore most notifications because 70% were false positives or informational noise. The remaining alerts are background static – they just tune out.
Here’s what that looks like in practice. A single poorly designed alert rule can drown a team in noise long before anything is actually wrong.
Bad alert: triggers constantly, no real signal
- alert: HighMemoryUsage
expr: container_memory_usage_bytes > 100000000
for: 1m
labels:
severity: warning
annotations:
summary: "Memory usage high"This rule fires any time a container nudges above an absolute 100 MB threshold for just one minute, regardless of its memory limit or role. In a cluster with hundreds of pods, that turns into dozens of meaningless alerts per hour, teaching engineers to ignore the very system that is supposed to help them.
This is less about tools piling up and more about clarity slipping away. What you’re missing hides behind a wall of alerts, even when screens glow with data. Like borrowed time, shortcuts today leave questions tomorrow. Each extra system promised relief, yet brought more tabs, noise, and effort just to trace one failure. Understanding gets heavier each time something new plugs a hole.
Observability Debt Builds Over Time
Most teams do not choose complexity all at once. It accumulates in layers: one tool for infrastructure, another for traces, a separate place for logs, and then more tooling added during migrations, reorganizations, or acquisitions. Each addition solves a local problem, but over time, the whole stack becomes harder to reason about, especially when incidents span multiple systems owned by different teams.
The Technical Manifestations
At the infrastructure monitoring level, each live time series consumes memory inside Prometheus’ head block, and that adds up quickly at scale. With 10 million active series, you’re looking at roughly 80 GiB of RAM just for in-memory indexing. That’s around 8 KiB per series as a practical provisioning baseline, though actual usage varies by label cardinality and scrape interval. Add multiple agents and data paths on top of that, and telemetry overhead starts competing with application workloads.
Operationally, this is where response quality drops. Logs sit in one system, metrics in another, traces in a third, and engineers stitch context together manually while the incident clock keeps running. New Relic’s 2024 study found organizations experience a median of 77 hours of annual downtime from high-impact IT outages, at a cost of up to $1.9 million per hour, and the data shows a strong correlation between full-stack observability and reduced downtime, meaning fragmented stacks pay a measurable price.
When telemetry is scattered across separate products, incident response turns into a copy‑and‑paste exercise instead of an investigation. A single latency spike in a payments service can easily look like this in practice:
# Step 1: pull logs from Splunk for a 2-minute window
index=prod service=payments earliest="2025-01-14 03:41:00" latest="2025-01-14 03:43:00"
# Step 2: check metrics for the same window
avg:payments.latency.p99{env:prod} by {host}.rollup(avg, 60)
# Step 3: find the matching trace in Jaeger
curl "http://jaeger:16686/api/traces?service=payments&start=1705196460000000&end=1705196580000000"
Three queries, three interfaces, one incident. If any of those clocks are a few seconds out of sync, you end up second‑guessing which spike, log line, or trace actually belongs together, and valuable minutes disappear into tool‑hopping instead of root‑cause analysis.
Left-behind dashboards pile up unused. Most displays have not drawn eyes in months, yet the searches feeding them still run. They burn through processing power and storage while showing nothing to anyone.
The Hidden Price Everyone Ignores
According to New Relic’s 2024 Observability Forecast, the median annual observability spend was $1.95 million, and organizations using a single tool spent 67% less than those using two or more tools: $700,000 versus $2 million.
In parallel, the Grafana 2025 Observability Survey found observability spend averages 17% of total compute infrastructure spend, though the median and mode both came in at 10%, with some organizations reporting upwards of 50%.
As a practical signal, if observability cost begins approaching infrastructure cost itself, the monitoring footprint likely needs consolidation rather than another tool.
What Consolidation Actually Delivers
Now things are shifting slowly. New Relic’s 2025 follow-up to its Observability Forecast shows organizations reducing the number of tools they use and moving toward consolidated platforms, with the average number of tools per organization dropping, and more than half of respondents planning to move to a unified system over the next two years. That change reflects a broader step back from scattered setups.
These days, most teams treat OpenTelemetry like standard equipment. It works through a single software kit and unified tools that send data wherever needed, with no need to modify apps each time. At its core, the OpenTelemetry Collector gathers signals in a way that ignores vendor boundaries, letting systems swap destinations freely. That shift matters because switching platforms used to mean rewriting instrumented code, a process that often took months.
One practical way to consolidate tools without risky big‑bang cutovers is to put the OpenTelemetry Collector in front of your current agents. It can receive telemetry once and fan it out to both your legacy backend and a new platform while you compare them side by side.
# otel-collector-config.yaml
# Receives OTLP data from your app, exports to two backends simultaneously
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
processors:
batch
exporters:
otlphttp/legacy:
endpoint: https://your-old-backend.example.com
otlphttp/new:
endpoint: https://your-new-platform.example.com
service:
pipelines:
traces:
receivers: [otlp]
processors: [batch]
exporters: [otlphttp/legacy, otlphttp/new]
With this setup, both backends receive the same traces while the migration runs. Once the new system checks out, you can remove the otlphttp/legacy exporter in a single config change, no application code touched, and no redeploys required.
Under the hood, eBPF tools tap directly into the kernel with minimal performance impact. Grafana Beyla, for example, adds around 0.5% CPU overhead under normal instrumentation conditions, rising to around 1.2% when application monitoring metrics are enabled. At Finastra, adopting eBPF-powered monitoring with Grafana Beyla reduced build failures from 20% to near zero, helped pinpoint critical timeout issues with only minor configuration changes, and prevented potential production outages, while lowering memory and infrastructure costs by consolidating observability agents.
Instead of paying the price during writes, tools such as ClickHouse handle heavy aggregation and computation at query time. This helps sidestep the memory pressure that hits Prometheus hard when labels multiply out of control. By shifting more work to query time, teams can often reduce infrastructure requirements and sharply cut the volume of metrics they need to store or export.
A Practical Approach to Combining Information
- Map Tools and Owners First: List each tool, what data it collects, and which team owns it. Mark overlap across logs, metrics, and traces, then prioritize duplicate capabilities for consolidation.
- Running a Monthly Orphan Audit: Using dashboard analytics, check which items were viewed during the past three months and remove those never seen and lacking an owner. When more than one in five alerts gets acknowledged, but nothing changes afterward, stop using that alert because it teaches staff to overlook warnings. Metrics not searched within a month should be removed without delay.
Here is a simple way to surface stale telemetry: query for metrics that have gone unused or have not received meaningful data over a fixed window.
# List series with zero samples in last 30 days
absent_over_time(up[30d])
Running this once usually uncovers a list longer than expected. Each entry is a storage, compute, and retention cost that survives every budget review because nobody thinks to check it.
- Consolidate in Sequence and Cap Spend: Consolidate logs first, then metrics, then traces. During migration, place the OpenTelemetry Collector ahead of existing agents so legacy and new backends run side by side. As a rule of thumb, when observability climbs past 15% of computing spend, investigate quickly; above 20%, debt is likely piling up.
Conclusion
Here’s how you get all debt fixed: start by halting new additions, then phase out what overlaps. Groups cutting expenses by nearly two-thirds while resolving issues far quicker aren’t drowning in extra information. They rely on a tighter set of systems to interpret identical inputs. Once monitoring charges climb near the price of running servers, the clutter turns into the very issue it promised to fix.
Frequently Asked Questions
What signs show you’re dealing with observability debt instead of having a solid monitoring setup?
Start by tallying your monitoring tools, then check where data streams repeat. When identical measurements pour in from three separate platforms tracking one host, duplication builds up. Imagine alerts lighting up. More than a fifth get marked, but nothing changes afterward. That’s noise piling on. When observability spend starts taking up an unusually large share of the infrastructure budget, it is worth checking whether overlapping tools and duplicated telemetry are driving unnecessary costs.
Is it possible to combine systems without tearing everything out first?
Sure. Run the OpenTelemetry Collector so it sends info to old systems at the same time as the new one. Check that the numbers look right and people are okay using it before switching fully. That way, progress shows step by step instead of betting everything on one move.
Cover Photo: Matthew Ansley on Unsplash