Welcome to the 2024 edition of the O11ys - the awards for the Observability industry. We launched the O11ys last year and the response was overwhelming, so it is a great pleasure to present the 2024 edition.
As we mentioned last year, the O11ys are not about one system being better than another and, as such, we don't have a category such as "Best Observability System". Instead, we endeavour to highlight particular contributions and achievements. Observability is an incredibly active and diverse space, with a truly spectacular rate of change. The O11ys therefore are our own idiosyncratic take, a snapshot of a particular moment.
For 2024 we have changed the format - with some new categories being added and some of last year's categories not being used this year. Without further ado, let's unveil the winners!'
Technology of The Year
Last year we saw the spectacular rise of eBPF, which opened up whole new vistas of technological possibilities by providing safe access to the Linux kernel. Whilst Iceberg does not claim a feat as daring as entering the inner sanctum of the Linux Kernel, it has nevertheless emerged as the undisputed champion of the storage wars, brushing aside multi-billion dollar competitors in the process.
The Perses project has set itself the ambitious target of defining open standards and specifications for dashboard creation and sharing. Over the past year it has made huge progress both in defining standards for dashboards as code as well as delivering resources such as the Perses server and SDKs. Perses is an Open Source project and was accepted into the CNCF as a Sandbox project in August 2024.
Best Newcomer
Quickwit has emerged onto the scene quietly but has quickly gained traction by posting incredible figures for performance and economy. The platform is able not only to ingest at petabyte scale but also query huge datasets at incredible speed and with a small resource footprint. Winning the custom of a hyperscaler such as Binance truly demonstrated the calibre of the product.
Best Use of AI
Many observability systems now claim to support Root Cause Analysis. At the same time though, most of these systems use algorithms - admittedly, advanced algorithms, which are based, fundamentally, on correlation rather than causation. For us Causely stands out as a system which truly embeds Causal AI in its reasoning and can therefore genuinely go beyond correlation and make more intelligent analysis of system behaviour.
As the year progressed, a number of vendors began to imbue their products with genuinely useful AI capabilities but the Logz.io approach really stood out. Their AI-driven assistant is clean and intuitive but also capable of delivering valuable and actionable insights.
Best Backend Database
A year ago, ClickHouse unveiled their vision for SQL-based observability and in the intervening 12 months they have made significant strides into turning that vision into a reality. They have been active on a number of fronts, with key updates to the database engine such as the introduction of a JSON primitive, as well as the development of the ClickHouse OpenTelemetry Exporter and a major integration with Grafana.
The Apache Software Foundation has established itself as a hotbed for data storage technologies with innovations such as Iceberg and Arrow. Another fruit of the Foundation's labours is Pinot, a powerful database platform that is used by a range of hyper-scalers including Uber, Slack, Wix and Etsy. In the observability space, it is also the backend for full-stack vendor Kloudfuse.
Best Mobile Observability Product
Embrace have pretty much written the book on mobile observability, filling what was a significant gap in the observability space. Over the past year they have chalked up some massive achievements - including open-sourcing their whole SDK and overcoming significant technical obstacles to provide full support for OpenTelemetry.
Embrace have been the leaders in the mobile space for a number of years but now other vendors are starting to realise the importance of dedicated mobile observability. In the past year both New Relic and Coralogix have rolled out mobile observability modules but the edge goes to New Relic. Their Mobile Monitoring product provides comprehensive end-to-end observability and deep insights across a range of platforms.
Best Front End Tooling
A number of vendors have recently added support for browser observability to their tooling, but we think that Honeycomb Front End is the most sophisticated and developer-friendly implementation we have seen. It goes way beyond covering the classic Core Web Vitals defined by Google and provides a coherent solution for end-to-end visibility with scope for unlimited custom attributes and sophisticated analytics.
Best LLM Observability Tooling
The explosive growth of LLM usage in application development has major implications for observability. Vendors need to do more than just tracing API calls, there are a whole raft of LLM-specific concerns that need to be monitored. A number of vendors have now incorporated LLM observability into their products, but we found the Datadog offering to be the most impressive. As well as core functionality such as monitoring LLM calls, it also provides sophisticated evaluations of response quality, cost analysis and scanning for sensitive data.
The Langtrace project only launched in March 2024 but it has developed at a breathtaking pace and has established itself as a leader in open source LLM observability. As well as being incredibly easy to set up and run it also has a large number of integrations with LLM Frameworks, Vector databases and observability backends.
Best Kubernetes Tooling
Kubernetes tooling is almost a market unto itself as there are a huge number of tools providing all sorts of functionality around the Kubernetes ecosystem. We have therefore split the category into two awards - one for Policy and Configuration and one for Troubleshooting and Monitoring.
Cilium is a flagship of the eBPF revolution. Despite its enormous power, it is surprisingly easy to get up and running and, combined with Hubble and Tetragon, represents a total solution for taking complete control of all traffic on a cluster. It presents engineers with almost unlimited possibilities for inspecting, managing and re-directing traffic flows.
Robusta is open source software that combines very powerful Kubernetes debugging capabilities with great ease of use and remarkable economy. If you spend a lot of time working with Kubernetes, you will really appreciate the user experience. The system has a rich UI incorporating an AI-powered chat assistant and maintains a laser-sharp focus on getting the job done quickly.
Best Developer Tooling
The phrase Observability Driven Development has started to gain some currency in recent times as vendors strive to overcome the disconnect between developers and traditional observability systems. Companies such as Kerno and Digma are leading the way in this movement, with tooling that integrates more closely with developer workflows. Kerno not only provides developers with observability signals, it also integrates with tooling such as Github to help developers identify and remediate issues seamlessly.
Digma is a slick and powerful tool that provides continuous feedback within the developer’s own IDE. The Digma engine continually analyses code and displays traces and analytics in the IDE in real time.
Best Cost Management Tooling
Akamas is quite simply an incredible product which delivers some extraordinary results. A lot of tools in the space take a policy-based approach to reducing costs, looking, for example, at using reserved instances or deallocating dormant resources. Akamas takes more of an engineering approach, seeking to reduce costs by hammering down compute. It does this by harnessing AI to analyse vast permutations of configurations and conducting experiments to evaluate the most efficient options and settings. This is a unique approach that delivers spectacular savings.
OpenCost is not just a leading cost management solution it is also open source and is a CNCF Incubating project. It is easy to use, supports a range of cloud providers and is under active development
Most Innovative New Product
The practice of Infrastructure as Code has led to vast improvements in system stability and consistency. At the same time, deployment scripts can be extremely complex and even the most carefully crafted scripts might have unforeseeable side-effects. Overmind claims to be the first tool that can conduct a 'pre-mortem' on your Terraform IaC scripts and warn you about changes that might break your AWS infrastructure. At the moment it only works with AWS but support for Azure, GCP, Pulumi and other providers is in the works.
Few new products made a more dramatic entrance into the marketplace than System Initiative. The launch blog proclaimed nothing less than a new paradigm in infrastructure provision - a revolutionary technology that would supercede Infrastructure as Code. Having tried the product out, we really like the idea of a drag and drop canvas for designing your infrastructure and think it offers a bold vision for moving beyond the constraints and frustrations of YAML-based IaC.
Best OpenTelemetry Implementation
The OpenTelemetry Collector is a pillar of the New Observability architecture. Despite its pivotal role though, in its native form it is still relatively primitive as it does not ship with a control plane or debugging features. Grafana were one of the first vendors to bring to market a solution that could help evolve the Collector from a simple Gateway to a manageable pipeline. Other vendors have followed suit but features such as Modules and Components have kept Alloy at the head of the pack.
Although Grafana were first to market with a wrapper for the oTel Collector, Datadog's oTel distribution is not only extremely powerful but was also the first to provide comprehensive support for fleet management.
News Story of the Year
At a time when the likes of Hashicorp, ScyllaDb and Redis are turning away from open source, the story of the year for us is the news of Elastic returning to the fold. It was a story that took most of us by complete surprise and was met with a certain amount of scepticism in some quarters. Strategically, adopting the same license that Grafana are now using may be a smart move and the company's commitment to the OpenTelemetry project will also serve to bolster its profile in the observability community.
When it comes to eye-catching feats of data processing, not many companies can beat ClickHouse. Whilst the One Billion Row Challenge provided some awesome feats of engineering, ClickHouse effortlessly took the game to the next level, polishing off the ingestion of a staggering one trillion rows in under three minutes.
Whilst it may not be in the same league as Cisco's acquisition of Splunk, Cloudflare's acquisition of Baselime also seemed to come right out of the blue. It is a great testimony to the quality of the product that Boris Tane and his associates built, but its assimilation as an internal Cloudflare product is an unfortunate loss to the marketplace.
Best Talk of The Year
It might be a hard sell convincing someone to watch a talk on Object Storage but this is a real masterclass from Justin Cormack, who is CTO at Docker. Object Storage has moved centre stage in observability engineering, and this is a really fascinating and informative exploration of the topic.
Best Blog
Bijit Ghosh is leading the Engineering and Architecture space in a top tier-1 bank. He has an incredible level of subject matter knowledge and blogs prolifically. The quality and depth of his writing is exceptional and his commitment to sharing his knowledge provides a fantastic service to the community.
Inevitably, a large volume of content in the blogosphere is now, at least partially, created by LLM's - so that we are swimming in a sea of increasingly homogenised content. It will be a while though, before an AI can catch up with Benn Stancil. Benn writes with a level of humour, verve and originality that may leave him as the last man standing when the rest of the bloggerati have succumbed to the relentless march of AI.
There are a number of eBPF-first vendors that have emerged offering products with zero-instrumentation and full stack coverage. Groundcover not only has excellent observability features, its hosting model also presents a compelling proposition in terms of cost and data governance.
Dash0 only launched relatively late in the year but are clearly a major new player in the observability space. They offer ease of use and power at a remarkably low cost. As well as having oTel support built in from the ground up, they also support Perses, which may reap dividends in the middle and longer term.
We first came across NetData via their presentation at the Open Source Observability Dayin October. What really stood out was an uncompromising commitment to performance and efficiency. The system already has a strong following, with over 72,000 GitHub stars and 650m DockerHub pulls.
Middleware are a full stack vendor that were founded in 2022 but underwent something of a reboot in 2024. They have built out their offering to include AI-powered alerts, RUM, Synthetic Monitoring and ingestion pipelines. With a rich feature set, petabyte scale ingestion and low costs they are offering a highly competitive proposition.
That concludes the O11ys for 2024. We really hope you have enjoyed our look back at the year. 2024 has been a pivotal, exciting and at times almost breathtaking year for observability. Existing vendors have expanded their portfolios, new vendors have emerged and a huge array of specialist products have entered the space. We look forward to sharing all the best in observability with you in 2025
Don't forget to also check out our review of 2024 where five of the leading figures in the observability space share their reflections on the year.
Happy New Year!
I am happy to disclose that I work as a Product Marketing Manager at SquaredUp, who are highly supportive of my work with Observability 360. At the same time, the O11ys are totally independent. There is no sponsorship and no vendor involvement in the selection process.
Even if you are not familiar with the name of Brendan Gregg, you are almost certainly familiar with the fruits of his labours. Brendan is the creator of the Flame Graph - one of the most important and iconic visualisations in the observability toolkit.
We featured the Flame Graph in our recent tribute to the work of UX designers in the observability arena - but you should also visit Brendans’ web site.
Brendan’s latest innovation is the AI Flame Chart. This is an evolution of the original flame graph and its ambitious aim is to help reduce the vast financial and environmental costs entailed in the use of LLM’s. This means that whereas the original flame graph was focused on CPU cycles, the latest generation sets its sights on reducing GPU load. The article discusses the considerable complexities involved in mapping GPU programs back to their corresponding CPU stacks.
The names of some of the instruction sets look intimidating to the uninitiated but the basic concept of the graph is quite simple - the wider the bar, the more resource it consumes.
If you have ever had to grapple with a 3,000 line Helm chart to deploy your observability infrastructure, you may be forgiven for thinking that there must be a better way to do this. Whilst YAML has a certain formal elegance, its syntax struggles to express the architectures and relationships embedded in highly complex systems.
Whilst Pulumi have tackled this problem by enabling the use of high level programming languages for IaC, System Initiative are taking a fundamentally more radical approach. Their goal is nothing other than completely reinventing IaC from the ground up. The blog article for the launch of the product is an incredibly ambitious statement of intent. The terms ‘game changer’ and ‘paradigm shift’ tend to be thrown around somewhat casually, this might be a case where their usage is appropriate.
So, what are they proposing? Well, System Initiative is IaC without the code. It is a kind of digital canvas where you manipulate digital twins of your systems. Is the future here or is this the Platform Engineering equivalent of science fiction? Read the article and decide for yourself!
Zomato is a restaurant aggregator and food delivery service that generates vast volumes of metrics. As their company grew, they adopted a Prometheus/Thanos-based architecture - running some 144 Prometheus servers. As metrics volumes continued to skyrocket, even this architecture started to creak and the Zomato SRE team began the search for an alternative solution.
In this article on the Zomato blog, the team discuss why they opted to migrate to Victoria Metrics as well as discussing a number of features of the system which enable them to achieve better performance, lower costs and greater scalability.
The technical challenges were pretty daunting - the project involved migrating over 800 dashboards, 300 microservices and 2.2 billion active time series. We would commend this article not just for its technical insights but also for taking a warts-and-all approach in documenting some of the technical limitations of the VM solution.
Grafana dashboards have been put to all sorts of uses over the years - for everything from space missions to monitoring milk production. In this fun but highly informative article Ivana Huckova and Sven Grossman walk us through building an observability system for bird song. Whilst this might sound slightly quirky, the techniques could be applied to all manner of applications which need to record and analyse audio inputs.
The article is a great showcase for a number of Grafana capabilities - including installing Alloy on a Raspberry Pi and adding context to Dashboard data by dynamically query sources such as Wikipaedia and the Open-Meteo weather information service.
Stories about Uber architecture always seem to be interesting, not least because they always involve technology at huge scale - such as this trillion record migration from DynamoDB. This article, however, is actually interesting on a number of levels. As well of being of technical interest it also provides some fascinating insight into internal team topologies and management processes - which are also fundamentally important aspects of managing observability at scale. Whilst most organisations will only operate at a fraction of Uber’s scale, every organisation is seeking to minimise costs and improve service to users, and the article provides a number of insights which would be of interest to most observability practitioners.
A survey carried out by McKinsey in 2021 found that 57% of respondents were already using Machine Learning to support at least one business function. ML is no longer a niche concern but is becoming a core component of development and CI/CD practices. As this post from the Datadog blog notes, the efficacy of ML models will inevitably degrade over time, so monitoring their performance and reliability is critical. The article really drives home the point that ML is a domain with its own specific behaviours, and effective monitoring requires building out new processes, metrics and even infrastructure to cover issues such as Data Drift, Prediction Drift and Concept Drift. Whilst the article does use some specialist terms, it is a highly readable and practical guide to the subject of ML monitoring.
It sounds like it could be a sub-plot in the film Inception, but this is a really interesting article from the Observe blog on how they use an instance of their Observe system to monitor their Observe cloud platform. Observe not only have to support fast reads for complex user queries, they also have to support ingesting one petabyte of telemetry per day. As you can see from the above diagram, Kafka and Snowflake form two of the pillars of the backend architecture. This three-part series offers a fascinating insight into Observe’s own internal observability strategy as well as being a great exemplar of the eat your own dog food principle. This is an article which is of great value to anybody with an interest in large-scale observability architectures.
Most of us have experienced the anguish of bill shock at some point. Being hit with a huge bill for mobile roaming charges on return from your holiday or getting a penalty notice for an inadvertent motoring infringement that happened weeks back. Those are just small pinpricks though, compared to the 50,000 volts of financial burn felt by companies mentioned in this transcript of a scintillating talk by Erik Peterson, CEO of CloudZero. He argues, persuasively, that engineering decisions are buying decisions. In the case mentioned in the headline, a decision to turn on one section of debug code led to vast volumes of logs being emitted and racking up over $1m in costs.
This is a really engaging blog post by Infrastructure Engineer Jack Lindamood, where he reviews nearly every infrastructure decision he made over four years working at a start-up. Each choice is graded with a Regret, Endorse or an occasional Unsure. Whilst not explicitly observability-related, it will however, have resonance for any engineer forced to make technological choices (which is probably all of us). The article contains much distilled wisdom and some strong opinions, as well as general observations on the challenges and trade-offs faced by infrastructure engineers.
The RAG pattern has really gained traction over the past year as it allows enterprises to leverage the power of LLM's to gain insights into their own data. This is a fascinating and (occasionally technical) article which details how Incident IO used vector embeddings to mine through their data and discover related incidents. The article explains the techniques involved with great clarity and provides really helpful advice on creating embeddings to find hidden patterns in your own data.
When you think of large corporations pushing the technology envelope, Chik-Fil-A might not be the first name to come to mind. However, the highly distributed nature of their infrastructure presents massive observability challenges, which they have met with some very impressive engineering. The scale of their task is daunting - 2,800 Edge Kubernetes clusters, tens of thousands of IoT devices and billions of MQTT messages each month. This is a really fascinating article on managing IoT observability at scale.
In this blog article, Bijit Ghosh of Deutsche Bank discusses best practices for observability across the full AI system lifecycle. He composes a custom system which knits together a range of technologies including structlog, Flask, Prometheus and Kibana as well as AI-specific tools such as MLFlow and CausalML. It’s a comprehensive article which exhibits a clear understanding of both observability and AI technologies.
A great example of managing the complexities of Observability engineering. Jay Taylor from InfluxDB builds out a solution using the Telegraf, InfluxDB, Grafana stack.
An in-depth look at monitoring K8S with the increasingly popular VictoriaMetrics platform. This follows an end-to-end process from crafting your own Helm chart to configuring alert rules.
This has really raised a few eyebrows. A forensic analysis by Nikolay Sivko of coroot on how just a few OpenTel meta tags can potentially explode your ingestion fees.
Comments on this Article