Logging
Stores
Store data where its readers are. Data that the app reads back, or joins with its own data, goes in the database. Events that people read, to debug or on a dashboard, go in the logs. Numbers over time go in metrics, and short-lived counters that the app reads go in the cache.
Database
A row in the database is for the app itself:
- The app reads it to decide something: the scheduled emails a mailer sends next, or the recent rejections a form checks before it accepts a submission.
- It belongs to a user or an object, and it goes when they go.
- Queries join it with other data, such as signup details next to whether the account was activated later.
- Losing it would be a bug, so a transaction protects it.
Rows with personal data need a retention period and a job that deletes them. An IP address is an online identifier that can identify a person (GDPR recital 30). Personal data may be kept no longer than its purpose needs, and only the fields that purpose needs (GDPR article 5). Leave session cookies and credentials out of every row.
Logs
Logs are events for people: requests, errors, warnings and runs of scheduled jobs. The app writes them to stdout. A log agent on the node collects them, and the log store keeps and deletes them: "a twelve-factor app never concerns itself with routing or storage of its output stream" (The Twelve-Factor App: logs). The app never reads its logs back.
Write one JSON object per line, so the log store can filter on fields. Loki's | json stage turns the properties of a JSON line into labels (Loki: log queries):
{namespace="example"} | json | level="WARNING"Logs last only as long as their storage. On a node, Kubernetes keeps the logs of one terminated container, and an evicted pod takes its logs with it (Kubernetes: logging architecture). A deploy replaces the pods, and kubectl logs only reaches the new ones. Without a cluster-level log store, anything a later decision depends on belongs in the database.
Metrics
Metrics are numbers over time: requests per second, rejections per reason, queue length. They feed dashboards and alerts, and they're cheap to keep for months. They aren't a record of single events: "if you need 100% accuracy, such as for per-request billing, Prometheus is not a good choice" (Prometheus: overview).
Cache
A counter that the app reads and that expires on its own, such as submissions per network this hour, goes in the cache. Losing it costs at most one period without the limit.
Bot protection
Bot protection on a signup form uses all four:
| Data | Store | Reader |
|---|---|---|
| A warning for each rejected submission | Logs | People who follow the logs |
| Every submission with its client signals | Database | Blocking rules and analysis |
| Submissions per network this hour | Cache | The rate limit |
| Rejections per reason over time | Metrics | Dashboards and alerts |