According to this study, native mobile applications account for nearly 90% of mobile device usage. This means that native apps are not only a key component of digital strategy but also that they need to be factored into overall observability workloads. This is especially true for markets such as India, where mobile accounts for over 75% of internet traffic.
This report, commissioned by New Relic, serves as a valuable reminder of the importance of high quality of service on mobile platforms. They found that slow load times and application freezes were a major turn-off for users - 60% of survey respondents said they wouldn’t stay on an app for more than 10 seconds if it showed slow buffer speeds.
Another report, commissioned by Embrace, highlights challenges facing app users and mobile engineers. Half of end-users said they experience some type of mobile app issue (crash, slow startup, unresponsive interface elements) on a daily basis. Mobile engineers reported that monitoring app performance is their top priority, both now and in the future.
The company is committed to the open source philosphy. They have recently open-sourced all of their SDK's as well as joining the CNCF. They also recently announced that all of their SDK's are now oTel compliant
Given their focus on mobile, they do not aim to be a full spectrum observability provider. Instead, they have developed partnerships with other players such as DataDog and Grafana whose technologies can complement the Embrace toolkit. This seems to chime with the Grafana philosophy that the observability space is an ecosystem rather than a zero-sum game.
Their pedigree is impressive - their technology team includes engineers previously in leading roles at the likes of New Relic, VMWare and GCP. In addition, the leadership team includes Alumni of DataDog, Apple and Google. The company's founders were previously involved in building large scale mobile businesses and their frustration at the lack of enterprise grade observability tooling was the primary motivation for creating Embrace. For example, CEO and co-founder Eric Futoran previously co-founded Scopely, the largest US mobile games publisher.
An ANR is a broken user experience caused by a prolonged app freeze. Getting to the root cause requires understanding the app conditions that led to the freeze. However, virtually every existing mobile monitoring solution only surfaces ANR data that Google exposes, which is a single stack trace taken several seconds after the app became frozen. The result is that mobile engineers waste time looking at snapshots that don’t provide actionable data.
In contrast, Embrace built a thread profiling feature specifically to profile apps as soon as they begin hanging. Mobile engineers can thus review the app state at the time of a freeze instead of several seconds later, which provides a more accurate picture of what actually caused the freeze. For mobile developers, dealing with these issues is not just important in terms of user experience. It is also critical for their application profile, as "badly behaving" apps will be penalised by Google and Apple. Mobile app developers also have to contend with the challenge that their apps will not only be running on different operating systems but also on an enormous variety of devices and networks. Most of these concerns have not really been catered by the existing solutions in the market.
Another major differentiator of the Embrace approach is that, by default, the runtime applies zero sampling at source. This means that clients operating in real-time observability mode can have instant visibility of any exceptions and apply immediate hotfixes before the issue spreads across the user base. Naturally, the flip side of this is that default retention periods are also short.
Application in your project.
If you don't have a class with that name, you will need to add it manually.
We found that the various dashboards offer useful overviews but for more detailed analytics you will need to use an external tool such as Grafana. This is not problematic however, as the Embrace API exposes a Prometheus endpoint and ships with a dashboard template populated with common queries. There are also integrations for exporting metrics to DataDog, New Relic and Honeycomb. You can then use these platforms for building more advanced custom queries e.g. total crashes broken down by app version/os/device model etc. Unfortunately, though, it seems as though custom metrics are only supported in the Enterprise version.
Embrace web site: https://embrace.io
Stories about Uber architecture always seem to be interesting, not least because they always involve technology at huge scale - such as this trillion record migration from DynamoDB. This article, however, is actually interesting on a number of levels. As well of being of technical interest it also provides some fascinating insight into internal team topologies and management processes - which are also fundamentally important aspects of managing observability at scale. Whilst most organisations will only operate at a fraction of Uber’s scale, every organisation is seeking to minimise costs and improve service to users, and the article provides a number of insights which would be of interest to most observability practitioners.
A survey carried out by McKinsey in 2021 found that 57% of respondents were already using Machine Learning to support at least one business function. ML is no longer a niche concern but is becoming a core component of development and CI/CD practices. As this post from the Datadog blog notes, the efficacy of ML models will inevitably degrade over time, so monitoring their performance and reliability is critical. The article really drives home the point that ML is a domain with its own specific behaviours, and effective monitoring requires building out new processes, metrics and even infrastructure to cover issues such as Data Drift, Prediction Drift and Concept Drift. Whilst the article does use some specialist terms, it is a highly readable and practical guide to the subject of ML monitoring.
It sounds like it could be a sub-plot in the film Inception, but this is a really interesting article from the Observe blog on how they use an instance of their Observe system to monitor their Observe cloud platform. Observe not only have to support fast reads for complex user queries, they also have to support ingesting one petabyte of telemetry per day. As you can see from the above diagram, Kafka and Snowflake form two of the pillars of the backend architecture. This three-part series offers a fascinating insight into Observe’s own internal observability strategy as well as being a great exemplar of the eat your own dog food principle. This is an article which is of great value to anybody with an interest in large-scale observability architectures.
Most of us have experienced the anguish of bill shock at some point. Being hit with a huge bill for mobile roaming charges on return from your holiday or getting a penalty notice for an inadvertent motoring infringement that happened weeks back. Those are just small pinpricks though, compared to the 50,000 volts of financial burn felt by companies mentioned in this transcript of a scintillating talk by Erik Peterson, CEO of CloudZero. He argues, persuasively, that engineering decisions are buying decisions. In the case mentioned in the headline, a decision to turn on one section of debug code led to vast volumes of logs being emitted and racking up over $1m in costs.
This is a really engaging blog post by Infrastructure Engineer Jack Lindamood, where he reviews nearly every infrastructure decision he made over four years working at a start-up. Each choice is graded with a Regret, Endorse or an occasional Unsure. Whilst not explicitly observability-related, it will however, have resonance for any engineer forced to make technological choices (which is probably all of us). The article contains much distilled wisdom and some strong opinions, as well as general observations on the challenges and trade-offs faced by infrastructure engineers.
The RAG pattern has really gained traction over the past year as it allows enterprises to leverage the power of LLM's to gain insights into their own data. This is a fascinating and (occasionally technical) article which details how Incident IO used vector embeddings to mine through their data and discover related incidents. The article explains the techniques involved with great clarity and provides really helpful advice on creating embeddings to find hidden patterns in your own data.
When you think of large corporations pushing the technology envelope, Chik-Fil-A might not be the first name to come to mind. However, the highly distributed nature of their infrastructure presents massive observability challenges, which they have met with some very impressive engineering. The scale of their task is daunting - 2,800 Edge Kubernetes clusters, tens of thousands of IoT devices and billions of MQTT messages each month. This is a really fascinating article on managing IoT observability at scale.
In this blog article, Bijit Ghosh of Deutsche Bank discusses best practices for observability across the full AI system lifecycle. He composes a custom system which knits together a range of technologies including structlog, Flask, Prometheus and Kibana as well as AI-specific tools such as MLFlow and CausalML. It’s a comprehensive article which exhibits a clear understanding of both observability and AI technologies.
A great example of managing the complexities of Observability engineering. Jay Taylor from InfluxDB builds out a solution using the Telegraf, InfluxDB, Grafana stack.
An in-depth look at monitoring K8S with the increasingly popular VictoriaMetrics platform. This follows an end-to-end process from crafting your own Helm chart to configuring alert rules.
This has really raised a few eyebrows. A forensic analysis by Nikolay Sivko of coroot on how just a few OpenTel meta tags can potentially explode your ingestion fees.
Comments on this Article