Up until quite recently, observability was pretty much synonymous with the proverbial "Three Pillars" of Logs, Metrics and Traces. Ingesting, storing and displaying these three signals was pretty much the definition of a Full Stack observability product. Naturally, for a number of years, top-end systems have gone beyond these basic requirements but essentially, supporting log aggregation, APM and dashboards meant you were covering all the bases.
Over the past year or so, though, it seems as though the table stakes have been raised. If you want to get into the game, then you are going to have to throw more functionality into the pot. Most likely this is the result of the increasingly competitive nature of the marketplace. The past few years have seen something of a Cambrian explosion in the observability space, with a host of new entrants coming into the market – GroundCover, Coroot, HyperDX and Betterstack to name but a few.
As vendors need to stand out and differentiate themselves from competitors, they will naturally add new features. The good news for customers is that these are actually features which add genuine value to the product, they are not just bells and whistles. There also appear to be different forces driving these innovations. On the one hand there are functional enhancements, driven by new or unfulfilled existing customer need. On the other hand, there are the new possibilities that have been opened up by advances in technology – particularly in the field of AI.
So, what are some of the features that will soon be bundled with the standard package? Well, first up is Anomaly Detection. Accurate and meaningful anomaly detection has been the Scarlet Pimpernel of observability - they seek it here, they seek it there, despite recent advances in machine-learning and AI though, it has remained highly elusive. This is partly due to the particular nature of Observability datasets. They tend to have highly irregular distributions and characteristics that do not lend themselves easily to many of the established anomaly detection algorithms. This naturally makes developing commoditised solutions difficult. If you are interested in finding out more, then this paper from Datadog does a good job of explaining some of the technical and mathematical complexities.
Despite these challenges, a number of vendors are now bringing solutions to the market. In recent months SigNoz, Victoria Metrics and Kloudfuse have all introduced Anomaly Detection tooling. The caveat though, is that this is not quite a magic bullet. Anomaly detection algorithms are not a one size fits all proposition. The Victoria Metrics implementation supports a number of different models. However, as their documentation points out, each model has its own assumptions - so you need to select the model which has the best fit with the profile of a particular dataset.
The next feature goes by a number of names – RUM, Front End Observability, Browser Observability – to mention a few. The ultimate goal though, is all about understanding and improving the user experience of your applications.
Arguably, for a long time, the focus of application observability has been on the back end – with an abundance of metrics for CPU usage, memory consumption, database latency, http 5xx errors etc. In an age of Single Page Applications and front end frameworks such as React and Angular, this means that a whole layer of application architecture has been neglected. In October, Honeycomb launched their Front End Observability tooling and vendors such as Kloudfuse, New Relic and Middleware have been busy rolling out or upgrading their own RUM functionality.
It is not always easy to tell exactly where we are with regard to the AI Hype Cycle. Certainly, some very big promises have been made – but we have not yet seen SRE’s being replaced by robots. Up until quite recently, the AI features in observability tooling were largely convenience functions such as automatically generating a title for a dashboard tile. Over the past few months however, a number of vendors have started to rollout some more substantial features, with a lot of effort being invested into reducing MTTR.
At one time, Root Cause Analysis meant a pretty laborious process of manual querying of logs and then cross-referencing with traces or even with logs etc from other sources. This is no longer good enough. Observability applications need to provide a more intelligent experience. They need to be able to detect failures and intelligently correlate telemetry across signals and across services.
We are now seeing this kind of functionality being rolled out by vendor after vendor. Grafana recently unveiled their Explore apps suite, which enables users to achieve insights into their telemetry without having to run queries. They have also rolled out a suite of AI-driven workflows for root cause analysis, built on the Asserts technology they acquired last year.
Logz.io are a company that have diligently held back from overplaying the possibilities of AI, but they have now also released their Observability IQ Assistant - which they describe as a "conversational AI" tool. Rather than searching through logs, users can ask questions such as "how is our service health" or "where do we need to improve performance". They have also rolled out their AI Agent which, they claim, can reduce Root Cause Analysis time by up to 70%. Similar capabilities are being developed or released by many more vendors in the space
Interestingly, this is not just happening in the observability mainstream. LLM API's such as Open AI, Llama and Claude are great levellers, enabling niche and point providers to compete on equal terms. Kerno and Causely, for example, ship with powerful root cause analysis features. Kerno can quickly identify the relevant line of code and even the relevant Git commit.
Root Cause Analysis is obviously just a diagnostic activity. A number of vendors are looking at taking the next step - automated incident resolution. Some vendors even talk about the ambitious goal of ‘closing the loop’ – i.e. a completely automated cycle of detection, identification and resolution without any human intervention. Whilst a number of specialist applications are already generating playbooks and scripts to resolve relatively routine issues it will require a quantum leap forward before most of us are willing to hand full control of our platforms to our friendly, neighbourhood robo-admins.
In this article I have just highlighted three of the areas in which vendors are now building added value into their products. There are also many more capabilities that vendors are offering as they move away from the three pillars baseline. Features such as LLM Observability, Incident Management, Mobile Observability and ingestion control planes are also appearing in product offerings. I am not suggesting that all of these capabilities will be essential in all applications across the board. It does seem, however, that the Three Pillars are now more of a foundational layer rather than representing the whole edifice of observability For some time, there has been an almost existential question mark hanging over observability. A lot of critics have asked what benefits observability tooling is actually bringing to justify the sometimes enormous costs involved. It seems to me that we are now at an inflection point where vendors are competing to add features that will bring real value to customers.
Zomato is a restaurant aggregator and food delivery service that generates vast volumes of metrics. As their company grew, they adopted a Prometheus/Thanos-based architecture - running some 144 Prometheus servers. As metrics volumes continued to skyrocket, even this architecture started to creak and the Zomato SRE team began the search for an alternative solution.
In this article on the Zomato blog, the team discuss why they opted to migrate to Victoria Metrics as well as discussing a number of features of the system which enable them to achieve better performance, lower costs and greater scalability.
The technical challenges were pretty daunting - the project involved migrating over 800 dashboards, 300 microservices and 2.2 billion active time series. We would commend this article not just for its technical insights but also for taking a warts-and-all approach in documenting some of the technical limitations of the VM solution.
Grafana dashboards have been put to all sorts of uses over the years - for everything from space missions to monitoring milk production. In this fun but highly informative article Ivana Huckova and Sven Grossman walk us through building an observability system for bird song. Whilst this might sound slightly quirky, the techniques could be applied to all manner of applications which need to record and analyse audio inputs.
The article is a great showcase for a number of Grafana capabilities - including installing Alloy on a Raspberry Pi and adding context to Dashboard data by dynamically query sources such as Wikipaedia and the Open-Meteo weather information service.
Stories about Uber architecture always seem to be interesting, not least because they always involve technology at huge scale - such as this trillion record migration from DynamoDB. This article, however, is actually interesting on a number of levels. As well of being of technical interest it also provides some fascinating insight into internal team topologies and management processes - which are also fundamentally important aspects of managing observability at scale. Whilst most organisations will only operate at a fraction of Uber’s scale, every organisation is seeking to minimise costs and improve service to users, and the article provides a number of insights which would be of interest to most observability practitioners.
A survey carried out by McKinsey in 2021 found that 57% of respondents were already using Machine Learning to support at least one business function. ML is no longer a niche concern but is becoming a core component of development and CI/CD practices. As this post from the Datadog blog notes, the efficacy of ML models will inevitably degrade over time, so monitoring their performance and reliability is critical. The article really drives home the point that ML is a domain with its own specific behaviours, and effective monitoring requires building out new processes, metrics and even infrastructure to cover issues such as Data Drift, Prediction Drift and Concept Drift. Whilst the article does use some specialist terms, it is a highly readable and practical guide to the subject of ML monitoring.
It sounds like it could be a sub-plot in the film Inception, but this is a really interesting article from the Observe blog on how they use an instance of their Observe system to monitor their Observe cloud platform. Observe not only have to support fast reads for complex user queries, they also have to support ingesting one petabyte of telemetry per day. As you can see from the above diagram, Kafka and Snowflake form two of the pillars of the backend architecture. This three-part series offers a fascinating insight into Observe’s own internal observability strategy as well as being a great exemplar of the eat your own dog food principle. This is an article which is of great value to anybody with an interest in large-scale observability architectures.
Most of us have experienced the anguish of bill shock at some point. Being hit with a huge bill for mobile roaming charges on return from your holiday or getting a penalty notice for an inadvertent motoring infringement that happened weeks back. Those are just small pinpricks though, compared to the 50,000 volts of financial burn felt by companies mentioned in this transcript of a scintillating talk by Erik Peterson, CEO of CloudZero. He argues, persuasively, that engineering decisions are buying decisions. In the case mentioned in the headline, a decision to turn on one section of debug code led to vast volumes of logs being emitted and racking up over $1m in costs.
This is a really engaging blog post by Infrastructure Engineer Jack Lindamood, where he reviews nearly every infrastructure decision he made over four years working at a start-up. Each choice is graded with a Regret, Endorse or an occasional Unsure. Whilst not explicitly observability-related, it will however, have resonance for any engineer forced to make technological choices (which is probably all of us). The article contains much distilled wisdom and some strong opinions, as well as general observations on the challenges and trade-offs faced by infrastructure engineers.
The RAG pattern has really gained traction over the past year as it allows enterprises to leverage the power of LLM's to gain insights into their own data. This is a fascinating and (occasionally technical) article which details how Incident IO used vector embeddings to mine through their data and discover related incidents. The article explains the techniques involved with great clarity and provides really helpful advice on creating embeddings to find hidden patterns in your own data.
When you think of large corporations pushing the technology envelope, Chik-Fil-A might not be the first name to come to mind. However, the highly distributed nature of their infrastructure presents massive observability challenges, which they have met with some very impressive engineering. The scale of their task is daunting - 2,800 Edge Kubernetes clusters, tens of thousands of IoT devices and billions of MQTT messages each month. This is a really fascinating article on managing IoT observability at scale.
In this blog article, Bijit Ghosh of Deutsche Bank discusses best practices for observability across the full AI system lifecycle. He composes a custom system which knits together a range of technologies including structlog, Flask, Prometheus and Kibana as well as AI-specific tools such as MLFlow and CausalML. It’s a comprehensive article which exhibits a clear understanding of both observability and AI technologies.
A great example of managing the complexities of Observability engineering. Jay Taylor from InfluxDB builds out a solution using the Telegraf, InfluxDB, Grafana stack.
An in-depth look at monitoring K8S with the increasingly popular VictoriaMetrics platform. This follows an end-to-end process from crafting your own Helm chart to configuring alert rules.
This has really raised a few eyebrows. A forensic analysis by Nikolay Sivko of coroot on how just a few OpenTel meta tags can potentially explode your ingestion fees.
Comments on this Article