Welcome to the Ollys - the Observability 360 end of year awards. This is our own individual and maybe slightly idiosyncratic take on the highlights of 2023. For example, we don't have a category such as "best Observability system". This is because the best observability system for a particular customer or user depends on a myriad of factors. Is Splunk a great system? Yes - and so are DataDog, Grafana, New Relic, SigNoz, Victoria Metrics, Honeycomb etc. As to which of those is the best - that is arguably in the eye of the beholder.
For this reason, the focus of our Ollys is on features, functions and technologies rather than on a particular stack. In addition to looking at systems and products we will also cast the net a bit wider and recognise actors across the broader observability community, who contribute so much via channels such as blogs, events, videos etc.
The selections presented here are, by their nature, subjective - if you have other nominations then please let us know in the comments or via Twitter. Also, if you have any ideas for additional categories, we would also love to hear your thoughts.
InfluxDB have really raised the bar with their V3 release - and get kudos for listening to user feedback in re-scoping Influx-QL. Scylla have dazzled this year but ClickHouse have really led the way with astonishing benchmarks and their vision for a SQL-based observability pipeline.
ScyllaDB has quickly gained a reputation as a rock solid solution for high availability and high throughput. It is growing quickly and counts the likes of Rakuten, Comcast and Expedia amongst its high profile users.
Cost management has been one of the hottest themes in observability in recent years, as the exponential increase in log volumes has led to huge spikes in ingress and storage costs. One cost-cutting option for cloud-based solutions is the use of spot-instances, and tooling such as Karpenter opens up this possibility with its intelligent management of K8S nodes. They have also won over major players such as Grafana, who announced that migrating to Karpenter had led to a 40% reduction in costs.
This is an elegant, lightweight logging tool that has a great UI and is packed with features. What's more it is open source.
Whilst AI promised much, eBPF delivered - opening up the Linux Kernel and paving the way for a number of solutions providing low-level, high-performance observability and networking. Cilium, Tetragon, Beyla and Pixie are just some of the tools which are pulling back the veil. Cisco's end of year acquisition of Isovalent really set the seal on a remarkable year.
This new platform combines the benefits of columnar data structures with in-memory computing to deliver high speed in-memory processing. It has already enabled OpenTelemetry to achieve a 40% improvement in compression efficiency for the majority of workloads.
For most of us, this came right out of the blue. Splunk is the observability behemoth, but the $29bn price tag was pretty eye-popping. Most financial analysts concluded that it was a smart move. Splunk's sophisticated SIEM software was a good fit for Cisco's customer base, who were demanding more advanced network security tooling. Splunk's strategic move to a cloud-based ARR model also offered the promise of a continuous and growing revenue stream.
This is a story that has pretty much entered into observability folklore and many column inches have been expended in discussing it. It has often, somewhat unfairly, been used as a stick with which to beat DataDog but, as this article notes, eight digit observability bills are nothing new.
Creating a film about a bunch of system enginers hacking the Linux kernel is a brave and magnificent proposition. This much trailed and eagerly awaited production did not disappoint - a riveting account of the long and sometimes arduous journey to build eBPF.
The scale of this gathering was truly impressive - 7,000 observability specialists gathered together under one roof for a vibrant and informative event.
Seifeddine Rajhi is a prodigious author who has produced a string of readable, authoritative and detailed investigations of a whole range of cloud technologies. One of the best bloggers on the Medium platform.
Many corporate blogs are merely thinly disguised adverts for their own products. Commendably, the SigNoz blog provides a rich stream of in-depth and highly professional articles on key topics of relevance to devs and observability engineers.
Danny Kopping of Grafana delivers a performance masterclass at the P99 conference. A fascinating investigation into the trade-offs involved in different storage and caching strategies and a surprisingly gripping tale.
A captivating presentation at RoachFest 2023 by Thomas Boltze of Santander. He covers the technical and cultural aspects of resilience with clarity and panache.
Some podcasts come across as people who like the sound of their own voice chatting into a mike for 30 minutes. The OpenObservability Talks podcasts really set the standard. They come across as well prepared and well structured and underpinned by a desire to inform and communicate. They have excellent guests and the topics are interesting and relevant.
There are a number of examples of products applying AI to certain features of their Stack but few companies have AI right at the core of their product as LogSail do. They are planning to rollout a major upgrade in 2024 which will feature anomaly prediction and detection.
Whilst both AWS and Splunk have released AI tools which use Natural Language processing to assist with search, both of these are still in preview as we write. Grafana have nudged ahead by rolling out AI-driven features designed to 'delight' the user. These are mainly aids for housekeeping - such as generating summaries and intelligent auto-completion - however, they represent significant steps in using AI to reduce toil for engineers.
Mobile is a huge but largely neglected space for observability. In 2023, Embrace emerged to fill this gap. Embrace is a new product but it is polished and feels highly mature. The Embrace toolkit provides SDK’s to integrate with all the major mobile development platforms and their API supports integrations with Grafana, DataDog, New Relic and other providers.
WebAssembly has the potential to revolutionise the way that applications are built, packaged and hosted. The intricacies of WebAssembly architecture however, can break conventional observability tooling. Dylibso are pioneering observability in this space.
SigNoz is a developer-oriented full stack observability offering with a focus on high performance and cost-effectiveness. They have a growing following, over 15 thousand stars on Github and bags of ambition.
VM is not just metrics, it is a full stack open source platform. It is gaining enormous traction and its metrics engine is emerging as a popular alternative to Prometheus.
The observability space has seen the arrival of a number of venture capital funded startups over the past few years. Chronosphere may be a relatively unknown player but they have some serious financial backing and their focus on reducing costs and avoiding data overload could be a winning formula.
Stories about Uber architecture always seem to be interesting, not least because they always involve technology at huge scale - such as this trillion record migration from DynamoDB. This article, however, is actually interesting on a number of levels. As well of being of technical interest it also provides some fascinating insight into internal team topologies and management processes - which are also fundamentally important aspects of managing observability at scale. Whilst most organisations will only operate at a fraction of Uber’s scale, every organisation is seeking to minimise costs and improve service to users, and the article provides a number of insights which would be of interest to most observability practitioners.
A survey carried out by McKinsey in 2021 found that 57% of respondents were already using Machine Learning to support at least one business function. ML is no longer a niche concern but is becoming a core component of development and CI/CD practices. As this post from the Datadog blog notes, the efficacy of ML models will inevitably degrade over time, so monitoring their performance and reliability is critical. The article really drives home the point that ML is a domain with its own specific behaviours, and effective monitoring requires building out new processes, metrics and even infrastructure to cover issues such as Data Drift, Prediction Drift and Concept Drift. Whilst the article does use some specialist terms, it is a highly readable and practical guide to the subject of ML monitoring.
It sounds like it could be a sub-plot in the film Inception, but this is a really interesting article from the Observe blog on how they use an instance of their Observe system to monitor their Observe cloud platform. Observe not only have to support fast reads for complex user queries, they also have to support ingesting one petabyte of telemetry per day. As you can see from the above diagram, Kafka and Snowflake form two of the pillars of the backend architecture. This three-part series offers a fascinating insight into Observe’s own internal observability strategy as well as being a great exemplar of the eat your own dog food principle. This is an article which is of great value to anybody with an interest in large-scale observability architectures.
Most of us have experienced the anguish of bill shock at some point. Being hit with a huge bill for mobile roaming charges on return from your holiday or getting a penalty notice for an inadvertent motoring infringement that happened weeks back. Those are just small pinpricks though, compared to the 50,000 volts of financial burn felt by companies mentioned in this transcript of a scintillating talk by Erik Peterson, CEO of CloudZero. He argues, persuasively, that engineering decisions are buying decisions. In the case mentioned in the headline, a decision to turn on one section of debug code led to vast volumes of logs being emitted and racking up over $1m in costs.
This is a really engaging blog post by Infrastructure Engineer Jack Lindamood, where he reviews nearly every infrastructure decision he made over four years working at a start-up. Each choice is graded with a Regret, Endorse or an occasional Unsure. Whilst not explicitly observability-related, it will however, have resonance for any engineer forced to make technological choices (which is probably all of us). The article contains much distilled wisdom and some strong opinions, as well as general observations on the challenges and trade-offs faced by infrastructure engineers.
The RAG pattern has really gained traction over the past year as it allows enterprises to leverage the power of LLM's to gain insights into their own data. This is a fascinating and (occasionally technical) article which details how Incident IO used vector embeddings to mine through their data and discover related incidents. The article explains the techniques involved with great clarity and provides really helpful advice on creating embeddings to find hidden patterns in your own data.
When you think of large corporations pushing the technology envelope, Chik-Fil-A might not be the first name to come to mind. However, the highly distributed nature of their infrastructure presents massive observability challenges, which they have met with some very impressive engineering. The scale of their task is daunting - 2,800 Edge Kubernetes clusters, tens of thousands of IoT devices and billions of MQTT messages each month. This is a really fascinating article on managing IoT observability at scale.
In this blog article, Bijit Ghosh of Deutsche Bank discusses best practices for observability across the full AI system lifecycle. He composes a custom system which knits together a range of technologies including structlog, Flask, Prometheus and Kibana as well as AI-specific tools such as MLFlow and CausalML. It’s a comprehensive article which exhibits a clear understanding of both observability and AI technologies.
A great example of managing the complexities of Observability engineering. Jay Taylor from InfluxDB builds out a solution using the Telegraf, InfluxDB, Grafana stack.
An in-depth look at monitoring K8S with the increasingly popular VictoriaMetrics platform. This follows an end-to-end process from crafting your own Helm chart to configuring alert rules.
This has really raised a few eyebrows. A forensic analysis by Nikolay Sivko of coroot on how just a few OpenTel meta tags can potentially explode your ingestion fees.
Comments on this Article