Observability on a Budget: Logs, Metrics, and Traces with Open-Source Tools

Observability bills have a reputation for surprising people. A few services, a forgotten debug log level, and one metric labeled with a user ID can turn monitoring into a real expense.

Open-source tools remove the license fees, but they add something else: your time.

This post covers what a small team needs first, a lean stack you can run yourself, the settings that control cost, and when to add each of the three signals: metrics, logs, and traces.

What you need first

The three signals answer different questions, and they are not equally expensive:

Metrics are numbers over time: request rate, error rate, latency, CPU. They tell you that something is wrong and power your alerts. They are usually the cheapest signal per unit of insight.
Logs are records of events. They tell you what happened. They are often the largest volume of data you produce.
Traces follow one request across services. They tell you where the time went. They matter most when many services call each other.

Start with metrics and a handful of alerts on the golden signals: latency, traffic, errors, and saturation. Add centralized logs next.

Add traces when you have a real question that metrics and logs can't answer, usually once requests cross several services.

Where the money goes

Signal	What drives the cost	Cheapest levers
Metrics	Cardinality: the number of unique label combinations	Bounded labels, longer scrape intervals, dropping unused metrics
Logs	Volume: how many lines you ship and keep	Drop debug logs, sample noisy lines, short retention
Traces	Span volume: every request can produce many spans	Sampling, keeping errors and slow requests

Two costs sit outside this table:

Retention: keeping data for longer is rarely free.
People: someone has to upgrade, secure, and monitor the stack.

Keep both in mind when you compare "free" tools with a hosted plan.

A lean reference stack

You can build a solid setup from a few well-known parts. Instrument your apps with OpenTelemetry, send most telemetry through one collector that acts as your cost-control point, and store each signal in a backend suited to it:

flowchart LR
    APP["Your apps<br/>OpenTelemetry SDKs"] --> COL["Collector<br/>OpenTelemetry Collector or Grafana Alloy<br/>filter, sample, batch"]
    HOST["Servers, containers,<br/>log files"] --> COL
    COL --> M[("Metrics<br/>Prometheus")]
    APP -. "scrape /metrics" .-> M
    COL --> L[("Logs<br/>Loki")]
    COL --> T[("Traces<br/>Tempo")]
    M --> G["Grafana<br/>dashboards and alerts"]
    L --> G
    T --> G

Metrics: Prometheus. It scrapes metrics endpoints and stores them locally, with a default retention of 15 days.

Since version 3.0 it can also receive OpenTelemetry metrics directly. That receiver is off by default (enable it with --web.enable-otlp-receiver) because Prometheus has no built-in authentication.

For high availability or long-term storage, look at Mimir, Thanos, or VictoriaMetrics.

Logs: Loki. Loki doesn't index the contents of your log lines. It indexes only a small set of labels per stream and stores the compressed log data in object storage such as S3, which keeps both the index and the bill small.

The trade-off is that searching log text is slower than in a system that indexes every word.

Traces: Tempo (or Jaeger). Tempo is Grafana's tracing backend. It stores traces in object storage without maintaining a separate index, which keeps costs low. You can skip it until you need it.

Dashboards and alerts: Grafana, with Prometheus Alertmanager (or Grafana's own alerting) for notifications.

Collection: the OpenTelemetry Collector or Grafana Alloy. Alloy is Grafana's distribution of the OpenTelemetry Collector.

Alloy also replaces Promtail, Loki's older log agent, which reached end of life on March 2, 2026. If a tutorial tells you to install Promtail, it is out of date.

A note on licensing: several Grafana Labs projects, including Loki, are licensed under AGPLv3. Running them for your own team is straightforward, but read the license before building a product or service on top of them.

Alternatives worth knowing

All-in-one platforms. SigNoz, OpenObserve, and HyperDX (acquired by ClickHouse) show logs, metrics, and traces in one interface and speak OpenTelemetry natively.
- SigNoz and HyperDX store data in ClickHouse, which is another system to run.
- OpenObserve is log-focused and ships as a single binary that can keep data in object storage.
- Licenses differ (OpenObserve, for example, is AGPL-3.0), so check each one before you commit.
The Victoria stack. VictoriaMetrics for metrics and VictoriaLogs for logs, paired with Grafana, is another option, often praised for low resource use and a simpler setup than the full Grafana stack.

There's no single winner. The right choice depends on how many moving parts you're willing to operate:

flowchart TD
    A["How many services and engineers?"]
    A -->|"One to three services, small team"| B["Metrics and logs first: Prometheus, Grafana, and Loki or VictoriaLogs"]
    A -->|"Many services calling each other"| C["Add traces: OpenTelemetry with Tempo, and sampling"]
    B --> D{"Prefer one product to several parts?"}
    C --> D
    D -->|Yes| E["All-in-one: SigNoz, OpenObserve, or HyperDX"]
    D -->|No| F["Grafana stack or Victoria stack, mixed as needed"]

Instrument once with OpenTelemetry

OpenTelemetry (OTel) is the vendor-neutral standard for generating telemetry. Its specification is stable for traces, metrics, and logs, and most backends accept its OTLP protocol.

The practical benefit for a budget is flexibility: instrument your code once and swap backends later without touching application code.

The Collector is where you control cost, because you can filter and sample before data reaches storage. Here is a trimmed example that drops debug logs and keeps all error and slow traces while sampling the rest:

receivers:
  otlp:
    protocols:
      grpc:
      http:
  # add more receivers here, such as filelog to tail log files

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 400
  filter/drop_debug_logs:
    error_mode: ignore
    log_conditions:   # older Collectors use logs: / log_record: instead
      - 'log.severity_number < SEVERITY_NUMBER_INFO'
  tail_sampling:
    decision_wait: 10s
    policies:
      - name: keep-errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: keep-slow
        type: latency
        latency: { threshold_ms: 500 }
      - name: sample-the-rest
        type: probabilistic
        probabilistic: { sampling_percentage: 10 }
  batch: {}

exporters:
  otlp/tempo:
    endpoint: tempo:4317
    tls:
      insecure: true    # only on a trusted private network
  otlphttp/loki:
    endpoint: http://loki:3100/otlp

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, tail_sampling, batch]
      exporters: [otlp/tempo]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, filter/drop_debug_logs, batch]
      exporters: [otlphttp/loki]

Loki 3.0 and later accepts OTLP logs directly, which is why the logs exporter points at Loki's /otlp endpoint. The older dedicated Loki exporter is deprecated.

Validate any Collector config against your version before relying on it, since processor options change between releases.

This example leaves out a metrics pipeline because Prometheus usually scrapes its targets. If your apps push OTLP metrics instead, send them to Prometheus's OTLP receiver, or use the Collector's prometheusremotewrite exporter (Prometheus then needs --web.enable-remote-write-receiver).

Metrics: keep cardinality in check

Every unique combination of label values creates a separate time series that Prometheus must store and index. High-cardinality labels are the main way metrics costs explode. Use labels with a small, bounded set of values:

# Bad: one time series per user and per order
http_requests_total{user_id="48213", path="/orders/9918"}

# Good: bounded labels
http_requests_total{route="/orders/:id", method="GET", status="200"}

A few more habits keep metrics cheap:

Scrape less often for anything that doesn't need fine resolution. Thirty or sixty seconds is plenty for most dashboards.
Drop metrics you never use at scrape time:

scrape_configs:
  - job_name: my-app
    scrape_interval: 30s
    metric_relabel_configs:
      - source_labels: [__name__]
        regex: "go_gc_duration_seconds.*"
        action: drop

Consider native histograms for latency. They are a more efficient alternative to classic histograms. They became stable in Prometheus 3.8, but scraping them is opt-in (scrape_native_histograms). Check that your client libraries and any long-term store support them too.
Keep default retention unless you need more. Fifteen days is enough for most day-to-day debugging.

Logs: fewer lines, better labels

Logs are often the most expensive signal, and much of what they hold is rarely read. Filter aggressively:

Drop or sample low-value lines, such as debug output and health checks, in production.
Write structured logs (JSON) and include the trace ID, so you can jump from a log line to its trace.
Keep Loki labels few and stable, like service, environment, and namespace. Labels define streams, and too many streams hurt performance.
Keep high-cardinality data out of labels. Put IDs in the log line itself, or in structured metadata (supported since Loki 3).
Use a short retention for verbose logs. A week or two is a reasonable starting point, and you can keep error logs longer.
Prefer Loki's monolithic deployment mode for a small setup. Its older "simple scalable" mode is deprecated and slated for removal in Loki 4.0.

Traces: add them when they pay off

For a single service or a monolith, metrics and good logs often answer most questions. Traces earn their keep when a request crosses several services and you need to see where the time went.

Because every request can generate many spans, you will sample. There are two approaches:

	Head sampling	Tail sampling
Decision made	At the start of a request	After the whole trace finishes
Cost	Cheap and simple	Needs memory to buffer traces
Weakness	May drop the one trace you needed	Needs all spans of a trace to reach the same Collector

Tail sampling lets you keep every error and slow trace and only a slice of routine ones, which is usually what you want. Three things to watch:

Memory. The Collector buffers whole traces until it decides.
decision_wait. Set it longer than your slowest traces.
Routing. Send all spans of a trace to the same instance. The Collector's load-balancing exporter exists for this.

Alerts: fewer and better

Alert on symptoms users feel (error rate, latency, saturation), not on every internal cause.
Page only for things a human must act on now. Send the rest to a ticket or a channel.
Monitor the monitor. Add an alert that always fires, routed to an external service that notifies you if the alerts stop arriving, and run an external uptime check. Otherwise a dead Prometheus looks exactly like a quiet night.

The hidden costs of self-hosting

Be honest about what "free" includes:

Upgrades and churn. Promtail's end of life is a good example: teams running it had to migrate to Alloy.
Storage growth and capacity planning, especially for logs.
Security. Prometheus ships without authentication, and Loki shouldn't be exposed to untrusted networks. Put your stack behind a proxy or private network.
On-call for your own tooling. When the observability stack is down, you are blind exactly when you need it.

For a very small team, a hosted free tier can cost less in total than self-hosting once you count your time. Compare real numbers before deciding.

A 30-day plan

Week one: run Prometheus and Grafana, build one dashboard of golden signals, set three to five symptom alerts, and add an external uptime check.
Week two: centralize logs with a small, stable label set, drop debug logs in production, and set retention.
Week three: instrument with OpenTelemetry and put a Collector in front of your backends.
Week four: add traces for your most important request path with tail sampling. Then review ingest volume and metric cardinality, and set a budget for each.

Wrapping up

Cheap observability is mostly about restraint. Collect fewer, better signals, and control cardinality and volume at the collector.

Use object storage where you can, add traces only when they answer real questions, and be realistic about the time your own stack will cost you. Start with metrics and alerts, and let real incidents tell you what to add next.