The market for logging tools is a pretty crowded space and new entrants need a distinctive proposition to make themselves stand out. LogSail is a startup from Charlotte, North Carolina and their product of the same name seeks to differentiate itself with a focus on AI and automation. The company’s CEO and founder, Stephen Collins took some time out of his hectic schedule to chat to us about the company and its product.
Stephen is probably one of the younger leaders in the observability industry, having progressed from an internship at Oracle to starting up in his own company in just a few years. At Oracle he worked on building distributed systems for automated application testing and had a lightbulb moment when he realised that the same design principles he used in those systems could be applied to processes such as log forwarding - which tend to be manually configured one at a time. Stephen realised there was potential for a product which could take the toil out of log configuration - and so LogSail was born.
The LogSail platform offers a cloud-based service which automates the process of log configuration. Its intelligent agents forward logs to the LogsailLogSail API and users can view and query logs via the web-based console. For any company in this space, optimising throughput via techniques such as filtering and compression is a must. As well as adopting these patterns LogSail also implements its own proprietary techniques to push the performance envelope. This includes taking advantage of MongoDB’s pluggable architecture to build out their own custom storage scheme. This is optimised for providing fast insertion and parallel search capabilities.
For those interested in diving deeper into implementation detail, each log collected by LogSail can be referenced in the storage tree structure using a four dimensional coordinate system. This results in a “logarithmic narrowing of the search pattern” and provides a foundation for horizontal scale-out and performant analytics.
One of the key selling points of the system is using automation to take the technical complexity out of log management. Indeed, you can be up and running in minutes by following a few simple steps. At the same time, where customers do have more complex needs, they can have direct access to engineers who can provide technical assistance with onboarding. Affording this level of customer support is one area where smaller companies such as LogSail can achieve a competitive advantage over some of the bigger players.
There are two fundamental parts to the application architecture. The first is a control plane where users define logging policies and logging sources. The second is AI-powered agents which autonomously identify log sources on the defined hosts. The agents use Reactive Machine learning to detect changes in local environments and then automatically re-set log configuration without any need for engineers to edit configuration files. This is a centrepiece of the LogSail vision - you will never need to edit another YAML file. The LogSail agents themselves periodically check for new logging sources or any updates to policies and will reconfigure themselves accordingly.
Defining policies and assigning them to hosts is a simple process which is managed in the LogSail web UI. Users can build queries to search for log events across multiple hosts, containers, apps, and clouds, and pull logs from Docker, systemd, and flat files. LogSail also integrates with Zeek, automatically discovering Zeek logs and forwarding them to its archival service with zero configuration.
The UI also offers the ability to pause logging at the touch of a button. This is a really powerful feature. Often, developers and sys admins either have to log on to a host and update config files or update application settings and redeploy in order to switch logs on or off. As well as avoiding the toil and risk involved in these interventions, the capability of turning off the tap on unwanted log streams can also be a valuable cost saving tool.
On the subject of cost, this is also an area where the company is highly competitive. They say that customers switching over from other providers can make savings of up 40%.
LogSail is not just about logs – it can also capture a wide range of system and networking metrics such as SSH, HTTP, and DNS traffic. This data can then be viewed alongside your logs in a consolidated platform. The discovery and autoconfiguration capabilities of LogSails make it a particularly good candidate for IoT and edge scenarios where potentially vast numbers of hosts may be in operation and new devices are continually being brought onstream or taken offline.
The company also has exciting plans for their next phase of development. Using a round of seed funding as a springboard, they are planning to rollout a major upgrade in 2024. This will use AI-powered analytics to sift through large volumes of log data to detect patterns which can be used for anomaly prediction and detection. They are also planning to launch Sevi, a LLM trained on individual customer source code, log data, deployments, and detected anomalies. Sevi will be able to provide level 1 support for DevOps and security teams investigating anomalous activity.
Whilst stories of huge bills for log storage often hit the headlines, Stephen observes that for many of the company's clients the swift remediation of incidents the is number one concern. The financial and reputational costs of production outages can be huge and can far outweigh storage and ingress costs - especially for companies big enough to negotiate discounts.
The vision of the company is a bold one – their goal is to “build a completely automated observability offer that responds to customer infrastructure/application changes and can perform root cause analysis automatically”. Even though they are a comparative fledgeling in the marketplace, they do have some big-hitters in their camp. Their list of advisors includes Kord Campbell – one of the founders of the highly respected Loggly platform (now part of SolarWinds), and Michael Cucchi – a VP at Sumo Logic. With the power of AI behind them and experienced hands at the helm, LogSail may have cause to be feeling buoyant.
You can find out more and even get started for free at the LogSail web site.
Stories about Uber architecture always seem to be interesting, not least because they always involve technology at huge scale - such as this trillion record migration from DynamoDB. This article, however, is actually interesting on a number of levels. As well of being of technical interest it also provides some fascinating insight into internal team topologies and management processes - which are also fundamentally important aspects of managing observability at scale. Whilst most organisations will only operate at a fraction of Uber’s scale, every organisation is seeking to minimise costs and improve service to users, and the article provides a number of insights which would be of interest to most observability practitioners.
A survey carried out by McKinsey in 2021 found that 57% of respondents were already using Machine Learning to support at least one business function. ML is no longer a niche concern but is becoming a core component of development and CI/CD practices. As this post from the Datadog blog notes, the efficacy of ML models will inevitably degrade over time, so monitoring their performance and reliability is critical. The article really drives home the point that ML is a domain with its own specific behaviours, and effective monitoring requires building out new processes, metrics and even infrastructure to cover issues such as Data Drift, Prediction Drift and Concept Drift. Whilst the article does use some specialist terms, it is a highly readable and practical guide to the subject of ML monitoring.
It sounds like it could be a sub-plot in the film Inception, but this is a really interesting article from the Observe blog on how they use an instance of their Observe system to monitor their Observe cloud platform. Observe not only have to support fast reads for complex user queries, they also have to support ingesting one petabyte of telemetry per day. As you can see from the above diagram, Kafka and Snowflake form two of the pillars of the backend architecture. This three-part series offers a fascinating insight into Observe’s own internal observability strategy as well as being a great exemplar of the eat your own dog food principle. This is an article which is of great value to anybody with an interest in large-scale observability architectures.
Most of us have experienced the anguish of bill shock at some point. Being hit with a huge bill for mobile roaming charges on return from your holiday or getting a penalty notice for an inadvertent motoring infringement that happened weeks back. Those are just small pinpricks though, compared to the 50,000 volts of financial burn felt by companies mentioned in this transcript of a scintillating talk by Erik Peterson, CEO of CloudZero. He argues, persuasively, that engineering decisions are buying decisions. In the case mentioned in the headline, a decision to turn on one section of debug code led to vast volumes of logs being emitted and racking up over $1m in costs.
This is a really engaging blog post by Infrastructure Engineer Jack Lindamood, where he reviews nearly every infrastructure decision he made over four years working at a start-up. Each choice is graded with a Regret, Endorse or an occasional Unsure. Whilst not explicitly observability-related, it will however, have resonance for any engineer forced to make technological choices (which is probably all of us). The article contains much distilled wisdom and some strong opinions, as well as general observations on the challenges and trade-offs faced by infrastructure engineers.
The RAG pattern has really gained traction over the past year as it allows enterprises to leverage the power of LLM's to gain insights into their own data. This is a fascinating and (occasionally technical) article which details how Incident IO used vector embeddings to mine through their data and discover related incidents. The article explains the techniques involved with great clarity and provides really helpful advice on creating embeddings to find hidden patterns in your own data.
When you think of large corporations pushing the technology envelope, Chik-Fil-A might not be the first name to come to mind. However, the highly distributed nature of their infrastructure presents massive observability challenges, which they have met with some very impressive engineering. The scale of their task is daunting - 2,800 Edge Kubernetes clusters, tens of thousands of IoT devices and billions of MQTT messages each month. This is a really fascinating article on managing IoT observability at scale.
In this blog article, Bijit Ghosh of Deutsche Bank discusses best practices for observability across the full AI system lifecycle. He composes a custom system which knits together a range of technologies including structlog, Flask, Prometheus and Kibana as well as AI-specific tools such as MLFlow and CausalML. It’s a comprehensive article which exhibits a clear understanding of both observability and AI technologies.
A great example of managing the complexities of Observability engineering. Jay Taylor from InfluxDB builds out a solution using the Telegraf, InfluxDB, Grafana stack.
An in-depth look at monitoring K8S with the increasingly popular VictoriaMetrics platform. This follows an end-to-end process from crafting your own Helm chart to configuring alert rules.
This has really raised a few eyebrows. A forensic analysis by Nikolay Sivko of coroot on how just a few OpenTel meta tags can potentially explode your ingestion fees.
Comments on this Article