Why Loki

Every machine writes logs, and logs are useless on the machine that wrote them. The moment a problem crosses two hosts, the only way to see it is to have the lines from both in one place, in order.

Drafted by an AI agent (claude-opus-5) from this lab’s own runbooks, deployment log and errata. Reviewed before publication by the site owner. How this site is written →

The fleet now ships its logs to one service. Getting there took a change of mind about the shipper and produced the most complete self-inflicted failure this project has recorded.

What it replaced

The Elastic stack is the default answer and has been for a decade: a search engine, a shipper, and a visualization layer, capable of far more than this lab needs.

Two things ruled it out. The resource cost is substantial, and it would have been the second heavyweight indexing system on a fleet that already runs one for security monitoring. And Elastic’s license change moved the core search engine off open source terms, which for a project whose whole premise is owning its own infrastructure is a consideration even when a fork exists.

Why this one

The deciding property is that Loki does not index the log contents.

Traditional log platforms parse every line and build a full-text index over all of it. That is what makes them powerful and what makes them expensive: the index is often larger than the logs, and the machine has to be sized for indexing rather than for storing.

Loki indexes only a small set of labels, essentially which host and which service a stream came from, and stores the log text compressed and unparsed. Queries filter by label first and then scan the matching chunks.

For a large organization searching arbitrary text across billions of lines, that is a real trade-off. For a lab where nearly every question starts with which machine and what service, it is not a trade-off at all. The labels are the query, and the part that was expensive was the part not being used.

The second reason is that the visualization layer was already there. Metrics and dashboards were being built on Grafana regardless, and logs land in the same interface next to the graphs rather than in a second system with its own login.

The shipper was swapped before it ever ran

The original design used Promtail, the log-shipping agent that had accompanied Loki for years. It was written into the machine baseline so every virtual machine would get it automatically.

It was replaced with Alloy before deployment, because Promtail had reached end of life. Its replacement is a broader collector that handles metrics, logs and traces in one agent rather than one agent per signal.

The interesting part is the timing. This was caught while checking that the chosen versions were current, not by deploying it and discovering the project had stopped. That check exists because of a rule this project arrived at the hard way: pin to current stable and treat vulnerability lists as a rejection floor, rather than pinning to the oldest version that fixes known problems. The identity provider note covers where that rule came from.

A version chosen carefully once still ages. The value of re-checking a pin before deployment is that it costs minutes at that moment and a migration afterwards.

How it is actually used here

Live, with the whole fleet shipping: the internal virtual machines, the rented gateways, and the physical servers underneath them.

The verification is worth describing because the first version of it was wrong. The question “are the hosts shipping logs” was answered for a while by checking whether the collector was running on each host. It was, everywhere. The answer was still wrong.

The right question is asked of the receiving end: which hosts has the log service actually heard from? That list is the only thing that distinguishes a collector that is running from a collector that is delivering, and the two came apart badly here.

What it cost

Six days of shipping into nothing. The collectors were deployed across the fleet before the service that receives their output existed. Every one of them started, reported itself active, and wrote errors to their own logs at a steady rate while delivering nothing.

The state was recorded honestly at the time, and then the note itself went stale: the receiving service was built, delivery started working, and the document went on saying it was broken for six days after it was fixed. The correction was made by measurement, querying the service for the hosts it had heard from, rather than by anyone remembering.

Both halves of that are the same lesson pointing in opposite directions. A claim that a thing is broken decays exactly like a claim that it works.

Label discipline is a design constraint, not a detail. The thing that makes this cheap is a small, bounded set of labels. Adding a label with many possible values, a request identifier, a timestamp, a user name, multiplies the number of distinct streams and undoes the property the whole choice was made for. The guard rail is that labels answer where did this come from, never what does it say.

It is not a search engine, and occasionally that is felt. A question of the form “find this string anywhere, across everything, over three months” is the case the design deliberately does not optimize for. That question is rare here and it is not free.

What would change the decision

The agent’s reasoning from the record rather than the human’s stated position.

  • If the questions stopped starting with a host. The entire argument is that label-first querying matches how this lab actually investigates. A workload where the useful question is full-text across everything would want the tool that indexes everything.
  • If log volume grew past what one machine should hold. The current arrangement is a single receiving service on a single virtual machine, which suits a fleet of this size and would need rethinking well before it became urgent.

About Loki

Loki is an open source log aggregation system developed by Grafana Labs, designed to index metadata labels rather than log contents, which keeps its storage and memory requirements substantially lower than full-text log platforms. It is queried through Grafana alongside metrics, using a query language deliberately similar to Prometheus. Logs are usually delivered by Alloy, Grafana’s collector for metrics, logs and traces. Both are released under the GNU Affero General Public License v3.