My title is Staff Site Reliability Engineer, and my role involves supporting and improving our observability tooling, enhancing incident response maturity, and working across the organization to help wherever needed, particularly in regards to keeping things running smoothly in production.
On any given day I might be investigating the root cause of an incident, troubleshooting build tooling, writing code, conducting a deep dive into some anomalous signal, or training engineers on how to interpret dashboards or metrics. I also help run a “metrics guild” which meets once a month where I prepare micro-learnings about using observability tools for monitoring and troubleshooting. I spend a lot of time thinking about ways to disseminate all the learnings from working with a diverse set of teams who encounter various challenges.
My role gives me a unique vantage point that spans infrastructure, platform services, application services, major changes or incidents, and all our tooling. This combination of contexts allows me to connect the dots and solve complex problems.
The first time I heard about the role of “Site Reliability Engineer” was a few jobs back when a VP approached me to bootstrap the company’s first SRE team. At the time, I was working as a Software Engineer and discovering the power of using New Relic to answer various ambiguous questions during production incidents. Suddenly, a team of three of us were tasked with being on call for the entire platform, requiring us to delve into unfamiliar territory. Learning how to leverage our telemetry signals - and eventually augment and extend them to make our system more observable - became a necessity for survival. Two years later, that VP would often brag about how our SRE team could get to the root cause of anything in “15 seconds” (just a slight exaggeration 😅). From there, I became addicted to the power that comes with knowing how to “read the tea leaves”.
In my current role, I primarily use Datadog for APM and metrics, and Splunk for logs. In previous roles, I used New Relic and ELK.
I have some exposure to Sentry for specific use cases, and I often experiment with open-source platforms like Elastic and SigNoz in small labs and personal projects.
I have this strange nostalgia for New Relic since it was the first tool I became intimately familiar with, and we went through so much together. I sometimes dream about writing NRQL queries… is that weird?
I believe it’s crucial to define a common interface for system components or services. For example, using common logging patterns, a core set of metrics, and consistent tracing conventions. Establishing these standards provides a solid baseline for observability.
In various positions, I've faced the frustration of navigating multiple similar tools used simultaneously and dealing with inconsistencies and duplication between them. For example, if both a Dropwizard metric registry and a Micrometer registry are used in the same Java application, the aggregation and reporting of timers or histograms can differ enough to confuse engineers who see different values for the same metrics.
The idea of unifying the approach to instrumentation through OpenTelemetry is very appealing to me. I have been closely monitoring this space and learning about it to identify when it might be an optimal solution for these types of problems.
We are only using OpenTelemetry for a few use cases currently, filling some small gaps in our other tools. However, I spend a significant amount of my personal time studying and contributing to various OpenTelemetry projects. I've always been fascinated by the tools I use daily, and I find that understanding how they work makes me a better operator.
Since OpenTelemetry is open source, it has allowed me to learn extensively about different implementations of instrumentation and new ways to approach observability. There are many smart and friendly people working on innovative projects in the community, and I enjoy following and learning from them.
I see OpenTelemetry as an exciting direction for the observability space, with significant momentum from many different vendors. The ability to decouple a bit from specific backends is a huge selling point. It feels like we're just at the beginning of its potential.
I spend a lot of time reading books, Reddit, and blogs, watching conference talks on YouTube, listening to occasional podcasts, and closely following several OpenTelemetry special interest groups. I watch or tune in to their weekly Zoom meetings and spend time hanging out in their slack channels. I learn best by doing, so I create small labs to experiment with new things or find ways to contribute to open source projects.
Every morning, I dedicate at least 25 minutes to a focused reading or study session, usually related to observability topics. For example, lately, I’ve been alternating between studying the OpenTelemetry documentation (and finding ways to contribute while I’m at it) and reading the new “Learning OpenTelemetry” book by Ted Young and Austin Parker.
Jay DeLuca is a Staff Site Reliability Engineer for Toast Inc, a provider of electronic services to the restaurant industry.
If you would like to feature in Observability People, please get in touch!
Stories about Uber architecture always seem to be interesting, not least because they always involve technology at huge scale - such as this trillion record migration from DynamoDB. This article, however, is actually interesting on a number of levels. As well of being of technical interest it also provides some fascinating insight into internal team topologies and management processes - which are also fundamentally important aspects of managing observability at scale. Whilst most organisations will only operate at a fraction of Uber’s scale, every organisation is seeking to minimise costs and improve service to users, and the article provides a number of insights which would be of interest to most observability practitioners.
A survey carried out by McKinsey in 2021 found that 57% of respondents were already using Machine Learning to support at least one business function. ML is no longer a niche concern but is becoming a core component of development and CI/CD practices. As this post from the Datadog blog notes, the efficacy of ML models will inevitably degrade over time, so monitoring their performance and reliability is critical. The article really drives home the point that ML is a domain with its own specific behaviours, and effective monitoring requires building out new processes, metrics and even infrastructure to cover issues such as Data Drift, Prediction Drift and Concept Drift. Whilst the article does use some specialist terms, it is a highly readable and practical guide to the subject of ML monitoring.
It sounds like it could be a sub-plot in the film Inception, but this is a really interesting article from the Observe blog on how they use an instance of their Observe system to monitor their Observe cloud platform. Observe not only have to support fast reads for complex user queries, they also have to support ingesting one petabyte of telemetry per day. As you can see from the above diagram, Kafka and Snowflake form two of the pillars of the backend architecture. This three-part series offers a fascinating insight into Observe’s own internal observability strategy as well as being a great exemplar of the eat your own dog food principle. This is an article which is of great value to anybody with an interest in large-scale observability architectures.
Most of us have experienced the anguish of bill shock at some point. Being hit with a huge bill for mobile roaming charges on return from your holiday or getting a penalty notice for an inadvertent motoring infringement that happened weeks back. Those are just small pinpricks though, compared to the 50,000 volts of financial burn felt by companies mentioned in this transcript of a scintillating talk by Erik Peterson, CEO of CloudZero. He argues, persuasively, that engineering decisions are buying decisions. In the case mentioned in the headline, a decision to turn on one section of debug code led to vast volumes of logs being emitted and racking up over $1m in costs.
This is a really engaging blog post by Infrastructure Engineer Jack Lindamood, where he reviews nearly every infrastructure decision he made over four years working at a start-up. Each choice is graded with a Regret, Endorse or an occasional Unsure. Whilst not explicitly observability-related, it will however, have resonance for any engineer forced to make technological choices (which is probably all of us). The article contains much distilled wisdom and some strong opinions, as well as general observations on the challenges and trade-offs faced by infrastructure engineers.
The RAG pattern has really gained traction over the past year as it allows enterprises to leverage the power of LLM's to gain insights into their own data. This is a fascinating and (occasionally technical) article which details how Incident IO used vector embeddings to mine through their data and discover related incidents. The article explains the techniques involved with great clarity and provides really helpful advice on creating embeddings to find hidden patterns in your own data.
When you think of large corporations pushing the technology envelope, Chik-Fil-A might not be the first name to come to mind. However, the highly distributed nature of their infrastructure presents massive observability challenges, which they have met with some very impressive engineering. The scale of their task is daunting - 2,800 Edge Kubernetes clusters, tens of thousands of IoT devices and billions of MQTT messages each month. This is a really fascinating article on managing IoT observability at scale.
In this blog article, Bijit Ghosh of Deutsche Bank discusses best practices for observability across the full AI system lifecycle. He composes a custom system which knits together a range of technologies including structlog, Flask, Prometheus and Kibana as well as AI-specific tools such as MLFlow and CausalML. It’s a comprehensive article which exhibits a clear understanding of both observability and AI technologies.
A great example of managing the complexities of Observability engineering. Jay Taylor from InfluxDB builds out a solution using the Telegraf, InfluxDB, Grafana stack.
An in-depth look at monitoring K8S with the increasingly popular VictoriaMetrics platform. This follows an end-to-end process from crafting your own Helm chart to configuring alert rules.
This has really raised a few eyebrows. A forensic analysis by Nikolay Sivko of coroot on how just a few OpenTel meta tags can potentially explode your ingestion fees.