Grey matter still matters. Maybe the best way to stem the tide of slop is to keep the human in the gloop.
There is a tendency to think that the major problems of observability are data problems or engineering problems or cost problems or maybe design problems. There is also a temptation to think that we can best serve users by reducing cognitive load - enacting the "Don't Make Me Think" mantra. So we throw AI and processing power and smart engineering and cheap storage into the mix. This can be effective but in some ways it is attacking the symptom rather than the problem.
Should observability really be about designing engineering solutions that brute-force their way through mountains of redundant telemetry in order to separate out the signals from the noise. Many systems today boast that they are capable of signal correlation - which is obviously a benefit. But maybe it begs the question of how the signals actually got silo-ed apart in the first place.
There are a few articles that have been published recently that point the way to a more holistic and preventive strategy for building observability systems. These are approaches that focus on crafting telemetry at the user end rather than processing telemetry at the vendor end. Rather than leaving observability vendors with the problem of processing excessive volumes of incoherent telemetry, maybe we could address the problem at source by getting smarter with instrumentation.
The first articles are part of a series being published by Juraci Paixão Kröhling. Juraci is an oTel contributor, former Grafana Engineer and the founder of OllyGarden. If you check out the OllyGarden web site and or read any of the blog articles you will get the sense that this is a project which has an ecological sensibility. The garden is not just about cultivating good telemetry for the sake of good engineering but also because all of those wasted CPU cycles do also burn CO2.
The premise of O11yGarden is that the problem of bad telemetry is one that should be tackled at the root. Having systems that can ingest and filter at scale is great, but what if we just create better quality and leaner telemetry in the first place. This then raises the question - what is bad telemetry? Well, this is where the OllyGarden really blossoms - it actually provides tooling for analysing your telemetry and then evaluating it against a set of objective criteria. It will then provide detailed feedback on on how you can improve your instrumentation. This has the potential to do for observability engineers what tools like Resharper did for software engineers. It is an assistant that helps you do better instrumentation. In doing so you have higher quality and more efficient telemetry.
In some ways, observability is a relatively young discipline. Computer programming has been around for several decades and there is a huge body of canonical knowledge on best practice. In observability, however, there is no Knuth. There is no Observability 101 on instrumentation. Yes, there are many, many engineers writing individual blogs and making great recommendations on particular tactics or tweaks but achieving best practice is often a matter of trial and error undertaken by individuals working in isolation.
The OllyGarden tooling changes all this. It is both a CT scan on your code and a mentor. It will analyse your instrumentation, it will detect 'code smells' and then it will suggest cleaner and more efficient alternatives. This addresses two potential sources of bad telemetry.
The first of these sources is badly crafted custom telemetry. This can take many shapes and forms. Many developers may never hand-craft a metric or a trace. For most of us, logging is as far as we go. In some ways, rolling your own telemetry is like rolling your own security. Yes, you can do it, but the potential for doing it badly is pretty huge. In security contexts, the result is that you create more vulnerabilities than you solve. In telemetry, it means that the code you craft to monitor issues can just create other issues. The obvious example if the proverbial cardinality explosion. As the OllyGarden team show, though, there are many other types of aberration that may spike your telemetry volumes and your costs.
The other source is seemingly more innocuous but may be still be problematic, and that is using instrumentation SDK's. For many of us, there is a temptation to think that auto-instrumentation with oTel and other instrumentation SDKs is a no-brainer and a bit of a free lunch. Unfortunately, this involves an element of wishful thinking. Yes, these packages are created by very smart people, but they inevitably make assumptions and work on averages and generalised use cases which may not be relevant or performant for your application.
The problem is that the majority of SME's do not have engineers who have both the time and skills to make sure that instrumentation is optimised and customised to the requirements of their application. The beauty of OllyGarden is not that it will throttle or divert your telemetry to a cheaper backend. Instead, it will support you in writing better telemetry.
The second major contribution to the Intelligent Observability approach was this article by Austin Parker on the OpenTelemetry blog. Again, this is quite a short piece and it is not screaming in block capitals with a click-baity title like "Logging - you are doing it wrong!!!". At the same time though. When you digest the article, the thought you may be left with is "LOGGING - AM I DOING IT WRONG??!!".
How can this be? Surely it is hard to get logging wrong? Surely everybody should be grateful that I am even bothering to manually log my code, especially if I am going above and beyond the call of duty by actually logging key business events rather than just logging my exceptions.
Yes, if you are logging key events and transactions that is cool. That is great. If you are using structured logging - even better. However, if you really want to take your logging to the next level, then maybe we can return to our earlier point about the separation of signals. The point here is that many events which are important enough to be logged do not occur discretely or in isolation. Often, they occur within the context of a transaction. In that case, why not attach those logs to that context at their point of creation. In general, this means embedding those logs within a trace.
Again, this is a practice which is, for the present time at least, one which requires an injection of human intelligence. It requires familiarity with the business logic of your code and making intelligent decisions about the nature of your logging. You need to understand which events are important and what are the important attributes you need to gather. Unlike the problem that OllyGarden is addressing, this does not necessarily mean that you need technical knowledge of the most efficient way to construct a histogram, but it does mean that you need to have knowledge of your business processes and form opinions about what information is important.
This then obviates the need for correlation. The signals are already combined. This is an argument that Honeycomb have been making for many years in their advocacy of the "wide event". This has always been an attractive concept but given that it has been mostly pushed by Honeycomb there has been a certain hesitancy around embracing it. Is it genuinely useful or is it just the preference of a particular vendor that just happens to work with their particular data structures.
Well, they are now not the only advocates of this model, ClickHouse have also thrown their weight behind this approach in their article for the launch of the ClickStack solution. There seems to be an increasing consensus that traces are the golden signal of distributed service instrumentation and that they are the natural container for many log events that were previously just generated as discrete and disconnected entities that need to be subsequently pulled together into a debugging context.
I started off this piece by saying "it's not about AI". I guess I should qualify this with a "As far as I know" or "Not yet". Who knows, given that OllyGarden are developing some formal systems for measuring and improving telemetry quality, there is obviously the possibility that before long machines can learn and apply these rules. For the moment though, the human is very much part of this semantic loop.
It’s an announcement that might have seemed unthinkable not long ago, but the porting of the revolutionary eBPF technology to Windows is now a reality. The ability to bring safe programmability to the kernel has resulted in enormous gains in fields such as security, networking and observability for Linux hosts, so applying the same principle to the Windows ecosystem is obviously an attractive proposition. It is not, though, without its own difficulties. There were a lot of hurdles to overcome and, inevitably, given the differences in OS architecture, this is not a full-fidelity replica of the Linux implementation.
This possibly foundational article by Pavel Yosifovich guides you through the steps involved in boldly going where few have gone before and creating your first eBPF program for Windows. One paragraph in the article begins with the sentence “this is where things get a bit hairy“ - for some that will likely be a challenge rather than a deterrent. This may not be cooking up nuclear fusion in your bedroom, but it does feel pretty radical.
As well as rolling out their Open AI observability solution, Elastic have also been very active within the OpenTelemetry project. C++ has a reputation for being something of a fearsome foe for observability practitioners. In this article on the Elastic blog, Haidar Braimaanie dons his protective gear and attempts to tame the beast with a soothing dose of OpenTelemetry instrumentation.
Unlike languages built in frameworks such as .NET, C++ does not have a standardized runtime environment that supports dynamic instrumentation across all platforms and compilers. C++ also uses a variety of build systems such as Makefiles and CMake, so that implementing instrumentation can be difficult and error-prone. In the article, Haidar looks at adding OpenTelemetry support to a C++ application running on Ubuntu 22.04. He also includes sample code for instrumenting the project with database spans and then observing the application in APM.
After reading this article you may want to give the C++ developer in your life a hug.
Even if you are not familiar with the name of Brendan Gregg, you are almost certainly familiar with the fruits of his labours. Brendan is the creator of the Flame Graph - one of the most important and iconic visualisations in the observability toolkit.
We featured the Flame Graph in our recent tribute to the work of UX designers in the observability arena - but you should also visit Brendans’ web site.
Brendan’s latest innovation is the AI Flame Chart. This is an evolution of the original flame graph and its ambitious aim is to help reduce the vast financial and environmental costs entailed in the use of LLM’s. This means that whereas the original flame graph was focused on CPU cycles, the latest generation sets its sights on reducing GPU load. The article discusses the considerable complexities involved in mapping GPU programs back to their corresponding CPU stacks.
The names of some of the instruction sets look intimidating to the uninitiated but the basic concept of the graph is quite simple - the wider the bar, the more resource it consumes.
If you have ever had to grapple with a 3,000 line Helm chart to deploy your observability infrastructure, you may be forgiven for thinking that there must be a better way to do this. Whilst YAML has a certain formal elegance, its syntax struggles to express the architectures and relationships embedded in highly complex systems.
Whilst Pulumi have tackled this problem by enabling the use of high level programming languages for IaC, System Initiative are taking a fundamentally more radical approach. Their goal is nothing other than completely reinventing IaC from the ground up. The blog article for the launch of the product is an incredibly ambitious statement of intent. The terms ‘game changer’ and ‘paradigm shift’ tend to be thrown around somewhat casually, this might be a case where their usage is appropriate.
So, what are they proposing? Well, System Initiative is IaC without the code. It is a kind of digital canvas where you manipulate digital twins of your systems. Is the future here or is this the Platform Engineering equivalent of science fiction? Read the article and decide for yourself!
Zomato is a restaurant aggregator and food delivery service that generates vast volumes of metrics. As their company grew, they adopted a Prometheus/Thanos-based architecture - running some 144 Prometheus servers. As metrics volumes continued to skyrocket, even this architecture started to creak and the Zomato SRE team began the search for an alternative solution.
In this article on the Zomato blog, the team discuss why they opted to migrate to Victoria Metrics as well as discussing a number of features of the system which enable them to achieve better performance, lower costs and greater scalability.
The technical challenges were pretty daunting - the project involved migrating over 800 dashboards, 300 microservices and 2.2 billion active time series. We would commend this article not just for its technical insights but also for taking a warts-and-all approach in documenting some of the technical limitations of the VM solution.
Grafana dashboards have been put to all sorts of uses over the years - for everything from space missions to monitoring milk production. In this fun but highly informative article Ivana Huckova and Sven Grossman walk us through building an observability system for bird song. Whilst this might sound slightly quirky, the techniques could be applied to all manner of applications which need to record and analyse audio inputs.
The article is a great showcase for a number of Grafana capabilities - including installing Alloy on a Raspberry Pi and adding context to Dashboard data by dynamically query sources such as Wikipaedia and the Open-Meteo weather information service.
Stories about Uber architecture always seem to be interesting, not least because they always involve technology at huge scale - such as this trillion record migration from DynamoDB. This article, however, is actually interesting on a number of levels. As well of being of technical interest it also provides some fascinating insight into internal team topologies and management processes - which are also fundamentally important aspects of managing observability at scale. Whilst most organisations will only operate at a fraction of Uber’s scale, every organisation is seeking to minimise costs and improve service to users, and the article provides a number of insights which would be of interest to most observability practitioners.
A survey carried out by McKinsey in 2021 found that 57% of respondents were already using Machine Learning to support at least one business function. ML is no longer a niche concern but is becoming a core component of development and CI/CD practices. As this post from the Datadog blog notes, the efficacy of ML models will inevitably degrade over time, so monitoring their performance and reliability is critical. The article really drives home the point that ML is a domain with its own specific behaviours, and effective monitoring requires building out new processes, metrics and even infrastructure to cover issues such as Data Drift, Prediction Drift and Concept Drift. Whilst the article does use some specialist terms, it is a highly readable and practical guide to the subject of ML monitoring.
It sounds like it could be a sub-plot in the film Inception, but this is a really interesting article from the Observe blog on how they use an instance of their Observe system to monitor their Observe cloud platform. Observe not only have to support fast reads for complex user queries, they also have to support ingesting one petabyte of telemetry per day. As you can see from the above diagram, Kafka and Snowflake form two of the pillars of the backend architecture. This three-part series offers a fascinating insight into Observe’s own internal observability strategy as well as being a great exemplar of the eat your own dog food principle. This is an article which is of great value to anybody with an interest in large-scale observability architectures.
Most of us have experienced the anguish of bill shock at some point. Being hit with a huge bill for mobile roaming charges on return from your holiday or getting a penalty notice for an inadvertent motoring infringement that happened weeks back. Those are just small pinpricks though, compared to the 50,000 volts of financial burn felt by companies mentioned in this transcript of a scintillating talk by Erik Peterson, CEO of CloudZero. He argues, persuasively, that engineering decisions are buying decisions. In the case mentioned in the headline, a decision to turn on one section of debug code led to vast volumes of logs being emitted and racking up over $1m in costs.
This is a really engaging blog post by Infrastructure Engineer Jack Lindamood, where he reviews nearly every infrastructure decision he made over four years working at a start-up. Each choice is graded with a Regret, Endorse or an occasional Unsure. Whilst not explicitly observability-related, it will however, have resonance for any engineer forced to make technological choices (which is probably all of us). The article contains much distilled wisdom and some strong opinions, as well as general observations on the challenges and trade-offs faced by infrastructure engineers.
The RAG pattern has really gained traction over the past year as it allows enterprises to leverage the power of LLM's to gain insights into their own data. This is a fascinating and (occasionally technical) article which details how Incident IO used vector embeddings to mine through their data and discover related incidents. The article explains the techniques involved with great clarity and provides really helpful advice on creating embeddings to find hidden patterns in your own data.
When you think of large corporations pushing the technology envelope, Chik-Fil-A might not be the first name to come to mind. However, the highly distributed nature of their infrastructure presents massive observability challenges, which they have met with some very impressive engineering. The scale of their task is daunting - 2,800 Edge Kubernetes clusters, tens of thousands of IoT devices and billions of MQTT messages each month. This is a really fascinating article on managing IoT observability at scale.
In this blog article, Bijit Ghosh of Deutsche Bank discusses best practices for observability across the full AI system lifecycle. He composes a custom system which knits together a range of technologies including structlog, Flask, Prometheus and Kibana as well as AI-specific tools such as MLFlow and CausalML. It’s a comprehensive article which exhibits a clear understanding of both observability and AI technologies.
A great example of managing the complexities of Observability engineering. Jay Taylor from InfluxDB builds out a solution using the Telegraf, InfluxDB, Grafana stack.
An in-depth look at monitoring K8S with the increasingly popular VictoriaMetrics platform. This follows an end-to-end process from crafting your own Helm chart to configuring alert rules.
This has really raised a few eyebrows. A forensic analysis by Nikolay Sivko of coroot on how just a few OpenTel meta tags can potentially explode your ingestion fees.
Comments on this Article