The observability space is a dynamic and heterogeneous realm. It is a habitat where commercial giants such as Datadog, New Relic and Splunk co-exist alongside the big OS/Freemium stacks such as ELK, Prometheus/Loki/Grafana and TIG (Telegraf/InfluxDB/Grafana). In recent years though, three technological trends have coalesced to reshape the landscape and pave the way for new stacks built on a new generation of powerful open source technologies and standards. The three new elements in the observability periodic table are:
Each of these are independent elements with totally different characteristics and functions but a new breed of observability contenders are assembling them into powerful frameworks.
OpenTelemetry is the second most active project in the CNCF (behind Kubernetes) and is the most significant project in observability today. Its principal benefit is to bring a lingua franca to the pre-existing Tower of Babel of competing protocols and semantics. It has also, however, unleashed two additional side-effects. Firstly, it empowers customers by freeing them from vendor lock-in. Secondly, it is a powerful enabler for new entrants into the market. OpenTelemetry compliance is attractive to consumers and new entrants to the market can enjoy the advantage of adopting it natively, whereas incumbents have to retro-fit or re-engineer existing codebases. OpenTelemetry also currently provides SDK's in 12 languages - thus obviating much of the pain and effort in writing your own agents. In addition to this, with a few Helm commands, you can deploy the OpenTelemetry Collector - providing you with a robust and immensely powerful gateway for ingestion of telemetry.
eBPFThe second element is eBPF. Opening up the Linux Kernel may seem like an esoteric techie endeavour. Its impact, however, has been seismic. It follows that if applications can safely access the kernel then they can have complete visibility of processes. From this stream of visibility flows the raw material for constructing traces and calculating metrics directly. To be fair, transforming that raw material into telemetry (and doing so without an excessive footprint) is far from trivial. At the same time though, the building blocks for that transformation can also be found in many open source repositories. Being able to tap into those repositories and build zero-instrumentation products endows observability vendors with game-changing capabilities.
The third element is ClickHouse, the alien-tech of our brave new observability architecture. It has touched down like an escapee from Lockheed Skunkworks that can achieve faster than light speeds. Meanwhile, its compression algorithms warp the fabric of diskspace so that it is bigger on the inside than on the outside. Ok - maybe that is a bit over the top. However, ClickHouse has emerged as the backend database for a remarkable number of disruptors and startups - Groundcover, SigNoz and DeepFlow to name but a few. Each time the rationale is the same. The ClickHouse engine is open source, can ingest at vast scale and query at speed. Its ultra-efficient compression also means cost savings which can be passed on to customers. As it''s open source there are also no database license fees.
A number of savvy and agile startups such as OneUptime, KloudMate and Groundcover have realized that these three components provide cost-free building blocks for assembling extremely powerful, robust and cost-effective systems - we might call it the OeC stack. All of the heavy lifting of storage, capture and structuring of telemetry are already pre-built. In the Open Source Hotel, there is such a thing as a free lunch. In fact, with OpenTelemetry, eBPF and ClickHouse you can feast on a free breakfast, lunch and dinner. Obviously, you cannot get these components to work together without sophisticated engineering and domain knowledge - however, the barriers to market have been drastically lowered.
With the advent of "cloud-native" computing, it is even possible to exclude OpenTelemetry and run on a slimmed down eBPF/ClickHouse stack. Obviously, the term "cloud native" is somewhat nebulous and is used very loosely as a marketing term. Effectively, it tends to mean running microservices in cloud-hosted K8S clusters. If your application follows this pattern then, theoretically, an eBPF-driven engine can generate all your traces and metrics with zero instrumentation. This is the premise of Cloud Native solutions such as Groundcover, DeepFlow and Coroot (although Coroot does also support OpenTelemetry for logs and traces). There are also other products currently in stealth mode that are adopting this architecture.
In the real world, it is unlikely that many systems will have such a conveniently clean and simple topology. Most distributed systems will also make calls to serverless apps, third party API's and messaging systems. Having requests originate outside of the boundaries of K8S is not necessarily a deal-breaker though. Products such as Grafana Beyla are capable of appending spans to incoming traces, so that the telemetry chain is not broken.
An increasing number of vendors are questioning the LMT (Logs, Metrics, Traces) paradigm and pulling away into a post-instrumentation era. Even though Observability thought leaders at companies such as Honeycomb natively support OpenTelemetry, they have been vocal about the fragmented nature of the current paradigm and advocated for a pattern based on "arbitrarily-wide structured log events". Perhaps occupying conceptually similar ground is the approach taken by vendors such as Observe and Dynatrace, who ingest multiple telemetry types into huge data lakes for combined analysis.
These trends lead to a nagging existential question around the whole OpenTelemetry project. It is, as we have said earlier, a magnificent edifice. But what if two of its three main pillars become redundant? Does it suffer the same fate as other meticulously constructed endeavours which are suddenly rendered obsolete by an evolutionary leap. The answer is probably not. Logs are still an essential pillar of observability and providers will still need to ingest them somehow. Even here, though, Groundcover's solution requires zero logging configuration. It's eBPF-powered Alligator agent will ingest logs automatically and forward them to an internal OpenTelemetry Collector. Equally, APM is not the only purpose of metrics collection - the use case of collecting device metrics for infrastructure, IoT etc monitoring and diagnostics will not be going away.
In this listing we will look at vendors who have adopted at least two of the three elements of the stack. Obviously, many companies support OpenTelemetry but those listed here rely on it as their principal ingestion mechanism.
| oTel | eBPF | ClickHouse | |
|---|---|---|---|
| SigNoz | ✓ | ✓ | |
| OneUptime | ✓ | ✓ | ✓ |
| Groundcover | ✓ | ✓ | ✓ |
| KloudMate | ✓ | ✓ | ✓ |
| Coroot | ✓ | ✓ | ✓ |
| DeepFlow | ✓ | ✓ | ✓ |
| HyperDX | ✓ | ✓ |
In reality, the likelihood is that OeC/eC is not going to replace the LMT paradigm or make OpenTelemetry obsolete any time soon. It is more likely indicative of the fact that observability is a rapidly evolving space and a growing market. Observability is in a state of flux - and is facing ever greater demands - it is being applied across more technologies, more architectures, and more domains. It is also being asked to provide richer insights and lower costs. As the market enlarges there is increasing space for niche players to enter the market with newer technologies and architectures aimed at solving more specific problems. This is because observability is not a zero-sum game - it is a growing ecosystem where many paradigms and technologies may coexist, integrate and complement one another.
Stories about Uber architecture always seem to be interesting, not least because they always involve technology at huge scale - such as this trillion record migration from DynamoDB. This article, however, is actually interesting on a number of levels. As well of being of technical interest it also provides some fascinating insight into internal team topologies and management processes - which are also fundamentally important aspects of managing observability at scale. Whilst most organisations will only operate at a fraction of Uber’s scale, every organisation is seeking to minimise costs and improve service to users, and the article provides a number of insights which would be of interest to most observability practitioners.
A survey carried out by McKinsey in 2021 found that 57% of respondents were already using Machine Learning to support at least one business function. ML is no longer a niche concern but is becoming a core component of development and CI/CD practices. As this post from the Datadog blog notes, the efficacy of ML models will inevitably degrade over time, so monitoring their performance and reliability is critical. The article really drives home the point that ML is a domain with its own specific behaviours, and effective monitoring requires building out new processes, metrics and even infrastructure to cover issues such as Data Drift, Prediction Drift and Concept Drift. Whilst the article does use some specialist terms, it is a highly readable and practical guide to the subject of ML monitoring.
It sounds like it could be a sub-plot in the film Inception, but this is a really interesting article from the Observe blog on how they use an instance of their Observe system to monitor their Observe cloud platform. Observe not only have to support fast reads for complex user queries, they also have to support ingesting one petabyte of telemetry per day. As you can see from the above diagram, Kafka and Snowflake form two of the pillars of the backend architecture. This three-part series offers a fascinating insight into Observe’s own internal observability strategy as well as being a great exemplar of the eat your own dog food principle. This is an article which is of great value to anybody with an interest in large-scale observability architectures.
Most of us have experienced the anguish of bill shock at some point. Being hit with a huge bill for mobile roaming charges on return from your holiday or getting a penalty notice for an inadvertent motoring infringement that happened weeks back. Those are just small pinpricks though, compared to the 50,000 volts of financial burn felt by companies mentioned in this transcript of a scintillating talk by Erik Peterson, CEO of CloudZero. He argues, persuasively, that engineering decisions are buying decisions. In the case mentioned in the headline, a decision to turn on one section of debug code led to vast volumes of logs being emitted and racking up over $1m in costs.
This is a really engaging blog post by Infrastructure Engineer Jack Lindamood, where he reviews nearly every infrastructure decision he made over four years working at a start-up. Each choice is graded with a Regret, Endorse or an occasional Unsure. Whilst not explicitly observability-related, it will however, have resonance for any engineer forced to make technological choices (which is probably all of us). The article contains much distilled wisdom and some strong opinions, as well as general observations on the challenges and trade-offs faced by infrastructure engineers.
The RAG pattern has really gained traction over the past year as it allows enterprises to leverage the power of LLM's to gain insights into their own data. This is a fascinating and (occasionally technical) article which details how Incident IO used vector embeddings to mine through their data and discover related incidents. The article explains the techniques involved with great clarity and provides really helpful advice on creating embeddings to find hidden patterns in your own data.
When you think of large corporations pushing the technology envelope, Chik-Fil-A might not be the first name to come to mind. However, the highly distributed nature of their infrastructure presents massive observability challenges, which they have met with some very impressive engineering. The scale of their task is daunting - 2,800 Edge Kubernetes clusters, tens of thousands of IoT devices and billions of MQTT messages each month. This is a really fascinating article on managing IoT observability at scale.
In this blog article, Bijit Ghosh of Deutsche Bank discusses best practices for observability across the full AI system lifecycle. He composes a custom system which knits together a range of technologies including structlog, Flask, Prometheus and Kibana as well as AI-specific tools such as MLFlow and CausalML. It’s a comprehensive article which exhibits a clear understanding of both observability and AI technologies.
A great example of managing the complexities of Observability engineering. Jay Taylor from InfluxDB builds out a solution using the Telegraf, InfluxDB, Grafana stack.
An in-depth look at monitoring K8S with the increasingly popular VictoriaMetrics platform. This follows an end-to-end process from crafting your own Helm chart to configuring alert rules.
This has really raised a few eyebrows. A forensic analysis by Nikolay Sivko of coroot on how just a few OpenTel meta tags can potentially explode your ingestion fees.
Comments on this Article