2024 Has been a spectacular year for observability. The market continues to expand and the domain of observability itself has firmly established itself in the mainstream of IT. It is also a dynamic and exciting place to be. The rise of eBPF, the rapid pace of OpenTelemetry adoption and the AI revolution are having a transformative effect on the industry.
For our review of 2024 we have invited four leading figures in the field to share their thoughts on the year in observability and, if they are feeling brave, make some predictions for the year ahead. The contributors themselves represent the remarkable breadth and diversity in the observability space at the moment. Andrew Tunall is President of Embrace - who are re-defining Mobile observability, whilst Sawaram Suthar is Founding Director of Middleware, one of the modern full stack vendors we referenced in our recent article on the changing face of observability. Diana Todea is a regular speaker at conferences and has recently been involved with Developing the new OpenTelemetry certification. Carly Richmond is well known for her advocacy role at Elastic - who are blazing a trail with the development of their AI platform.
Diana Todea is a Senior SRE at EQS. She is a regular speaker at observability conferences and is a contributor to the OpenTelemetry project.
Reflecting on the year 2024's trends and events, we cannot ignore AI's mark on observability. Many of the demos I have seen focused on how AI Assistants can improve incident management and DevOps/SRE workflows and we also witnessed the release of a lot of new vector databases.
Out of the 14 tech conferences I have participated in this year, one stood out for me: the OSMC conference in Nuremberg, Germany. I never thought I would see so many good OSS alternatives to the commercial observability platforms. One particular demo from Netdata opened my eyes: seeing an entire o11y platform embed with ML workflows and taking the developer experience to a whole new level. It made me think I definitely want to test it more but also made me wonder: Will these "magical" ML/observability platforms make our SRE/DevOps engineer jobs irrelevant? Maybe the saying "AI will not steal your job, someone who knows AI will" still holds water. My advice to you for 2025 is: stay open, stay sharp and always keep learning!
Andrew Tunall is President and CPO of Embrace and was previously VP of Product at New Relic
As digital interactions become increasingly complex, users demand seamless performance and intuitive interfaces. I’ve been hearing about digital transformation for 10 years, but transformations to how frontend teams build and operate web and mobile apps have been lagging behind. 2025 seems poised to be the year where the consumer shift that has been happening during that period is finally reflected in how the market changes its observability focus and practices.
With support for distributed traces, metrics, and logs, OpenTelemetry unifies data collection, empowering teams to pinpoint performance bottlenecks or sluggish response times. But those core telemetry types only recently have started to be correlated in a way that tells the story about what user experiences look like.
What sets the future apart is the shift toward actionable insights. Observability will no longer be about amassing data; it will focus on translating telemetry into answers, whether through better correlations or AI summarization. And increasingly, those answers will be about user impact.
I’m excited to see the shift as more frontend engineering teams get engaged in the OpenTelemetry project, and observability in general, and what that means for better user experiences in the web and mobile apps we use in our everyday lives!
Carly Richmond is a Principal Developer Advocate at Elastic and her interests include OpenTelemetry and eBPF with a Sprinkling of AI
2024 for me has been an interesting year in Observability. OpenTelemetry and eBPF have both had an increased buzz about them. In conversations, I’m finding that logging and basic metrics are commonly used, but there is a divide in the adoption of tracing that separates intermediate and advanced practitioners. Conversely, some tools such as Synthetic Monitoring, specifically monitoring-as-code practices and OTel client instrumentation, are quietly simmering. You can’t always predict a tool's popularity.
Cost is still a key consideration, impacting data retention and cost. I speak to many engineers switching to free tooling or consolidating their tools. What’s important is to ensure migration efforts, productivity, training and visibility across your application ecosystem are maintained.
It would be rude not to mention AI adoption on many platforms. 2024 has been the year when many vendor platforms added assistants and other AI capabilities to help explain, diagnose, and potentially remediate issues. These tools have great power, but we need to validate the sources of information. For this reason, I don’t believe AI will replace junior DevOps Engineers, SREs, or Platform Engineers anytime soon.
I’m not brave enough to predict what will emerge in 2025. But I’m excited to continue the journey!
Sam Suthar is Founding Director at Middleware, a unified observability platform
In his contribution, Sam addresses two key trends shaping current observability practice and strategies.
Excessive data collection has led to inflated costs without real value. Customizable observability platforms now enable companies to flag and filter unnecessary data, reducing expenses without sacrificing insights. By implementing comprehensive data processing with compression and indexing, companies can significantly reduce data size, leading to cost savings. This approach is expected to save companies 60-80% on observability costs, shifting from exhaustive data collection to efficient, targeted monitoring.
AI is transforming observability by automating root cause analysis, anomaly detection, and predictive alerts. AI-powered platforms identify patterns, highlight potential issues, and suggest fixes, thereby enhancing efficiency and reducing mean time to resolution (MTTR). AI-driven anomaly detection capabilities enable the identification of unknown unknowns, detecting unusual patterns of behavior not seen before, and allowing for timely investigation and remediation. Furthermore, predictive alerting can even pinpoint specific telemetry data (logs, metrics, traces, or events) that may indicate issues, enabling proactive remediation. This level of automation and foresight is redefining the future of observability and reliability.
I would like to say a huge thanks to Diana, Sam, Andrew and Carly for taking time out of their schedules to share their thoughts and reflections. The Year In Observability Review is only the first instalment in our end of year coverage. On December 31st we will be rounding off the year in style with the unveiling of the 2024 O11ys - the Oscars for the observability industry. We ran the awards for the first time last year and they sparked enormous interest within the community. We can't wait to reveal the 2024 winners!'
Even if you are not familiar with the name of Brendan Gregg, you are almost certainly familiar with the fruits of his labours. Brendan is the creator of the Flame Graph - one of the most important and iconic visualisations in the observability toolkit.
We featured the Flame Graph in our recent tribute to the work of UX designers in the observability arena - but you should also visit Brendans’ web site.
Brendan’s latest innovation is the AI Flame Chart. This is an evolution of the original flame graph and its ambitious aim is to help reduce the vast financial and environmental costs entailed in the use of LLM’s. This means that whereas the original flame graph was focused on CPU cycles, the latest generation sets its sights on reducing GPU load. The article discusses the considerable complexities involved in mapping GPU programs back to their corresponding CPU stacks.
The names of some of the instruction sets look intimidating to the uninitiated but the basic concept of the graph is quite simple - the wider the bar, the more resource it consumes.
If you have ever had to grapple with a 3,000 line Helm chart to deploy your observability infrastructure, you may be forgiven for thinking that there must be a better way to do this. Whilst YAML has a certain formal elegance, its syntax struggles to express the architectures and relationships embedded in highly complex systems.
Whilst Pulumi have tackled this problem by enabling the use of high level programming languages for IaC, System Initiative are taking a fundamentally more radical approach. Their goal is nothing other than completely reinventing IaC from the ground up. The blog article for the launch of the product is an incredibly ambitious statement of intent. The terms ‘game changer’ and ‘paradigm shift’ tend to be thrown around somewhat casually, this might be a case where their usage is appropriate.
So, what are they proposing? Well, System Initiative is IaC without the code. It is a kind of digital canvas where you manipulate digital twins of your systems. Is the future here or is this the Platform Engineering equivalent of science fiction? Read the article and decide for yourself!
Zomato is a restaurant aggregator and food delivery service that generates vast volumes of metrics. As their company grew, they adopted a Prometheus/Thanos-based architecture - running some 144 Prometheus servers. As metrics volumes continued to skyrocket, even this architecture started to creak and the Zomato SRE team began the search for an alternative solution.
In this article on the Zomato blog, the team discuss why they opted to migrate to Victoria Metrics as well as discussing a number of features of the system which enable them to achieve better performance, lower costs and greater scalability.
The technical challenges were pretty daunting - the project involved migrating over 800 dashboards, 300 microservices and 2.2 billion active time series. We would commend this article not just for its technical insights but also for taking a warts-and-all approach in documenting some of the technical limitations of the VM solution.
Grafana dashboards have been put to all sorts of uses over the years - for everything from space missions to monitoring milk production. In this fun but highly informative article Ivana Huckova and Sven Grossman walk us through building an observability system for bird song. Whilst this might sound slightly quirky, the techniques could be applied to all manner of applications which need to record and analyse audio inputs.
The article is a great showcase for a number of Grafana capabilities - including installing Alloy on a Raspberry Pi and adding context to Dashboard data by dynamically query sources such as Wikipaedia and the Open-Meteo weather information service.
Stories about Uber architecture always seem to be interesting, not least because they always involve technology at huge scale - such as this trillion record migration from DynamoDB. This article, however, is actually interesting on a number of levels. As well of being of technical interest it also provides some fascinating insight into internal team topologies and management processes - which are also fundamentally important aspects of managing observability at scale. Whilst most organisations will only operate at a fraction of Uber’s scale, every organisation is seeking to minimise costs and improve service to users, and the article provides a number of insights which would be of interest to most observability practitioners.
A survey carried out by McKinsey in 2021 found that 57% of respondents were already using Machine Learning to support at least one business function. ML is no longer a niche concern but is becoming a core component of development and CI/CD practices. As this post from the Datadog blog notes, the efficacy of ML models will inevitably degrade over time, so monitoring their performance and reliability is critical. The article really drives home the point that ML is a domain with its own specific behaviours, and effective monitoring requires building out new processes, metrics and even infrastructure to cover issues such as Data Drift, Prediction Drift and Concept Drift. Whilst the article does use some specialist terms, it is a highly readable and practical guide to the subject of ML monitoring.
It sounds like it could be a sub-plot in the film Inception, but this is a really interesting article from the Observe blog on how they use an instance of their Observe system to monitor their Observe cloud platform. Observe not only have to support fast reads for complex user queries, they also have to support ingesting one petabyte of telemetry per day. As you can see from the above diagram, Kafka and Snowflake form two of the pillars of the backend architecture. This three-part series offers a fascinating insight into Observe’s own internal observability strategy as well as being a great exemplar of the eat your own dog food principle. This is an article which is of great value to anybody with an interest in large-scale observability architectures.
Most of us have experienced the anguish of bill shock at some point. Being hit with a huge bill for mobile roaming charges on return from your holiday or getting a penalty notice for an inadvertent motoring infringement that happened weeks back. Those are just small pinpricks though, compared to the 50,000 volts of financial burn felt by companies mentioned in this transcript of a scintillating talk by Erik Peterson, CEO of CloudZero. He argues, persuasively, that engineering decisions are buying decisions. In the case mentioned in the headline, a decision to turn on one section of debug code led to vast volumes of logs being emitted and racking up over $1m in costs.
This is a really engaging blog post by Infrastructure Engineer Jack Lindamood, where he reviews nearly every infrastructure decision he made over four years working at a start-up. Each choice is graded with a Regret, Endorse or an occasional Unsure. Whilst not explicitly observability-related, it will however, have resonance for any engineer forced to make technological choices (which is probably all of us). The article contains much distilled wisdom and some strong opinions, as well as general observations on the challenges and trade-offs faced by infrastructure engineers.
The RAG pattern has really gained traction over the past year as it allows enterprises to leverage the power of LLM's to gain insights into their own data. This is a fascinating and (occasionally technical) article which details how Incident IO used vector embeddings to mine through their data and discover related incidents. The article explains the techniques involved with great clarity and provides really helpful advice on creating embeddings to find hidden patterns in your own data.
When you think of large corporations pushing the technology envelope, Chik-Fil-A might not be the first name to come to mind. However, the highly distributed nature of their infrastructure presents massive observability challenges, which they have met with some very impressive engineering. The scale of their task is daunting - 2,800 Edge Kubernetes clusters, tens of thousands of IoT devices and billions of MQTT messages each month. This is a really fascinating article on managing IoT observability at scale.
In this blog article, Bijit Ghosh of Deutsche Bank discusses best practices for observability across the full AI system lifecycle. He composes a custom system which knits together a range of technologies including structlog, Flask, Prometheus and Kibana as well as AI-specific tools such as MLFlow and CausalML. It’s a comprehensive article which exhibits a clear understanding of both observability and AI technologies.
A great example of managing the complexities of Observability engineering. Jay Taylor from InfluxDB builds out a solution using the Telegraf, InfluxDB, Grafana stack.
An in-depth look at monitoring K8S with the increasingly popular VictoriaMetrics platform. This follows an end-to-end process from crafting your own Helm chart to configuring alert rules.
This has really raised a few eyebrows. A forensic analysis by Nikolay Sivko of coroot on how just a few OpenTel meta tags can potentially explode your ingestion fees.
Comments on this Article