
Observability at scale demands a unified approach to logs, metrics, and traces. The Grafana LGTM stack - Loki, Grafana, Tempo, and Mimir - provides a robust, cloud-native solution. This guide details its production deployment, focusing on OpenTelemetry (OTel) integration, cross-signal correlation, and cost optimization.
traceID label, or from a Mimir metric spike to a Tempo waterfall in a single click.limits_config.retention_period + S3 backend to keep hot indexes small while retaining raw logs cheaply in object storage.The LGTM stack, augmented by OpenTelemetry Collectors, forms a cohesive observability platform.
Key Components:
The OTel Collector is critical for standardizing telemetry data. We'll configure it to receive OTLP, process, and export to Mimir, Loki, and Tempo.
otel-collector-config.yaml)receivers:
otlp:
protocols:
grpc:
http:
processors:
batch:
send_batch_size: 1024
timeout: 10s
resource/add_service_name: # Ensure service.name is present for correlation
attributes:
- key: service.name
value: unknown_service
action: upsert
attributes/add_k8s_labels: # Example: Add Kubernetes labels as resource attributes
actions:
- key: k8s.pod.name
from_attribute: k8s.pod.name
action: upsert
- key: k8s.namespace.name
from_attribute: k8s.namespace.name
action: upsert
transform/logs: # Transform OTel logs to Loki-compatible format
log_statements:
- context: log
statements:
- set(attributes["loki.tenant.id"], "default") # Multi-tenancy example
- set(attributes["loki.resource.labels"], Concat([resource.attributes["service.name"], resource.attributes["k8s.namespace.name"]], ","))
# Ensure trace_id and span_id are available as attributes for Loki
- set(attributes["trace_id"], SpanIDToHex(trace_id))
- set(attributes["span_id"], SpanIDToHex(span_id))
transform/metrics: # Example: Add tenant ID to metrics
metric_statements:
- context: metric
statements:
- set(attributes["tenant_id"], "default")
resource/add_tenant_id: # Add tenant ID to resource attributes for traces
attributes:
- key: tenant_id
value: default
action: upsert
exporters:
otlp/mimir:
endpoint: mimir-gateway.observability.svc.cluster.local:9090 # Mimir OTLP endpoint
tls:
insecure: true # Use proper TLS in production
otlp/loki:
endpoint: loki-gateway.observability.svc.cluster.local:3100 # Loki OTLP endpoint
tls:
insecure: true
otlp/tempo:
endpoint: tempo-gateway.observability.svc.cluster.local:4317 # Tempo OTLP endpoint
tls:
insecure: true
logging: # For debugging
verbosity: detailed
service:
pipelines:
metrics:
receivers: [otlp]
processors: [resource/add_service_name, attributes/add_k8s_labels, transform/metrics, batch]
exporters: [otlp/mimir, logging]
logs:
receivers: [otlp]
processors: [resource/add_service_name, attributes/add_k8s_labels, transform/logs, batch]
exporters: [otlp/loki, logging]
traces:
receivers: [otlp]
processors: [resource/add_service_name, resource/add_tenant_id, batch]
exporters: [otlp/tempo, logging]
Explanation:
receivers.otlp: Configures the collector to accept OTLP data over gRPC and HTTP.processors.batch: Batches telemetry data for efficient export.processors.resource/add_service_name: Ensures service.name is always present, crucial for correlation.processors.attributes/add_k8s_labels: Demonstrates enriching telemetry with Kubernetes metadata, useful for filtering and grouping.processors.transform/logs: This is critical for Loki. It adds loki.tenant.id and loki.resource.labels (which Loki uses as indexed labels) and extracts trace_id and span_id into log attributes, enabling trace-to-log correlation.exporters.otlp/mimir, otlp/loki, otlp/tempo: Direct OTLP export to the respective LGTM components. Ensure these endpoints are resolvable within your cluster.service.pipelines: Defines how data flows through receivers, processors, and exporters for each signal type (metrics, logs, traces).Unified observability hinges on the ability to navigate between signals. Grafana facilitates this with data link configurations.
When viewing a trace in Tempo, you want to jump to logs associated with a specific span. This requires the trace_id and span_id to be present as attributes in your logs. The transform/logs processor in the OTel Collector handles this.
Grafana Data Link Configuration (Tempo Datasource):
{
"name": "Logs for Span",
"url": "/explore?orgId=1&left=[\"now-1h\",\"now\",\"Loki\",{\"expr\":\"{service_name=\\\"$${service.name}\\\", trace_id=\\\"$${traceID}\\\", span_id=\\\"$${spanID}\\\"}\",\"refId\":\"A\",\"queryType\":\"range\"},{\"ui\":true}]",
"targetBlank": true
}
Explanation:
$${service.name}: Inferred from the trace span.$${traceID}: The trace ID of the current span.$${spanID}: The span ID of the current span.expr uses Loki's LogQL to filter logs by service_name, trace_id, and span_id. These labels must be present in your Loki logs (as configured by the OTel Collector).From a log line in Loki, you want to jump to the corresponding trace in Tempo. This requires the trace_id to be present as an attribute in the log.
Grafana Data Link Configuration (Loki Datasource):
{
"name": "Trace for Log",
"url": "/explore?orgId=1&left=[\"now-1h\",\"now\",\"Tempo\",{\"query\":\"$${__line.trace_id}\",\"queryType\":\"search\",\"refId\":\"A\"},{\"ui\":true}]",
"targetBlank": true
}
Explanation:
$${__line.trace_id}: This special Grafana variable extracts the trace_id attribute from the current log line. This assumes your OTel Collector has added trace_id as an attribute to logs.Leveraging Mimir, we can build robust SLI/SLO dashboards. This example focuses on a request latency SLI.
Define recording rules in Mimir to pre-aggregate SLI data. This reduces query load on the raw metrics and improves dashboard performance.
# rules.yaml for Mimir
groups:
- name: service-sli
rules:
- record: service_latency_sli_total
expr: |
sum by (service_name, http_route) (
rate(http_server_request_duration_bucket{le="0.5"}[5m])
)
- record: service_latency_slo_error_budget
expr: |
1 - (
sum by (service_name, http_route) (
rate(http_server_request_duration_bucket{le="0.5"}[5m])
)
/
sum by (service_name, http_route) (
rate(http_server_request_duration_count[5m])
)
)
Explanation:
http_server_request_duration_bucket: Assumes your application exports OpenTelemetry HTTP server request duration histograms.service_latency_sli_total: Counts requests within the latency target.service_latency_slo_error_budget: Calculates the error budget consumed.(
sum by (service_name, http_route) (service_latency_sli_total)
/
sum by (service_name, http_route) (rate(http_server_request_duration_count[5m]))
) * 100
sum by (service_name, http_route) (
increase(service_latency_slo_error_budget[7d])
)
100 - (
sum by (service_name, http_route) (
increase(service_latency_slo_error_budget[7d])
)
)
Mimir, Loki, and Tempo heavily rely on object storage. Optimizing this is crucial for cost control.
Configure retention periods based on data criticality and compliance.
table_manager and compactor handle retention.compactor manages retention.Example Loki Retention Configuration (loki.yaml):
table_manager:
retention_period: 90d # 90 days for index and chunks
compactor:
retention_enabled: true
retention_period: 90d
Leverage cloud provider lifecycle policies to transition older data to cheaper storage tiers (e.g., S3 Standard-IA, Glacier, GCS Nearline, Coldline).
Example S3 Lifecycle Policy (Terraform aws_s3_bucket_lifecycle_configuration):
resource "aws_s3_bucket_lifecycle_configuration" "loki_bucket_lifecycle" {
bucket = aws_s3_bucket.loki_chunks.id
rule {
id = "loki-retention"
status = "Enabled"
transition {
days = 30
storage_class = "STANDARD_IA" # Infrequent Access
}
transition {
days = 60
storage_class = "GLACIER" # Archival
}
expiration {
days = 90 # Final deletion
}
}
}
All LGTM components use compression (Snappy, LZ4, Zstd) for data stored in object storage. Ensure your configurations are leveraging efficient codecs. This is typically default, but worth verifying.
max_chunk_age and chunk_target_size to balance write performance and query efficiency. A smaller max_chunk_age means more frequent flushing, potentially more small objects, but faster index updates.Symptom: OTel Collector pods are OOMKilled or show high CPU/memory usage, dropping telemetry data.
Cause: Ingest rate exceeds processing/export capacity, or batch processor settings are too aggressive for available memory.
Fix:
batch processor: Increase send_batch_size and timeout to allow larger batches, but monitor memory.memory_limiter processor: Configure a soft and hard limit to prevent OOMs and gracefully drop data under extreme pressure.
processors:
memory_limiter:
limit_mib: 256
spike_limit_mib: 64
check_interval: 1s
Symptom: LogQL queries are slow, especially those involving regex or large time ranges. Cause:
line_format to extract data from log content instead of indexing it.query-frontend and querier components.max_chunk_age and chunk_target_size are balanced.Symptom: Traces are missing or incomplete in Tempo. Cause:
curl with traceparent headers to test.max_block_bytes: If traces are very large, this might be a bottleneck.Symptom: Mimir ingesters/compactors are struggling, high storage costs, slow metric queries. Cause: Too many unique label combinations on metrics. Fix:
metricstransform processor or Prometheus relabel_configs to drop or rename high-cardinality labels before ingestion.
processors:
metricstransform:
transforms:
- include: "http_request_duration_seconds_bucket"
action: "delete_label"
label: "request_id" # Example: delete high-cardinality label
max_series_per_user and max_label_names_per_series to prevent abuse.In a production LGTM stack deployment, an engineer observes high CPU utilization on the OpenTelemetry Collector instances during peak traffic. Which of the following is the most effective strategy to mitigate this, considering the goal of maintaining comprehensive observability without significantly increasing operational costs?
A1: Mimir, Loki, and Tempo are designed for multi-tenancy.
X-Scope-OrgID HTTP header (or OTLP attribute tenant_id) to separate tenants. Each tenant gets isolated data.X-Scope-OrgID or the loki.tenant.id OTLP log attribute.X-Scope-OrgID or the tenant_id OTLP resource attribute.X-Scope-OrgID in dashboards.A2: Helm charts are the standard. Grafana Labs provides official charts for Loki, Mimir, and Tempo. For the OpenTelemetry Collector, use the opentelemetry-collector Helm chart. Ensure you configure persistent storage for stateful components (e.g., Mimir ingesters, Loki ingesters) and use a robust object storage backend.
A3: The LGTM components expose Prometheus metrics. Deploy a dedicated Prometheus instance (or use Mimir itself) to scrape these metrics. Create Grafana dashboards to monitor their health, resource usage, ingest rates, query latencies, and error rates. Key metrics include loki_ingester_received_entries_total, mimir_ingester_received_samples_total, tempo_ingester_traces_received_total, and various _grpc_server_handled_total metrics.
A4: Yes. Mimir is a long-term storage for Prometheus. You can configure your existing Prometheus servers to remote-write to Mimir, or use the OpenTelemetry Collector's Prometheus receiver and OTLP exporter to Mimir. The latter is generally preferred for a unified OTel approach.
A5:
| Feature | Loki (Label-based) | Traditional (Full-text) |
|---|---|---|
| Cost | Lower storage, less compute for indexing | Higher storage, more compute for indexing |
| Query Speed | Fast for label-filtered queries, slower for text | Fast for full-text searches, slower for complex filters |
| Scalability | Highly scalable horizontally | Scalable, but index management can be complex |
| Flexibility | Requires pre-defined labels for efficient search | Searchable on any log content |
| Use Case | Operational debugging, known patterns | Security analysis, ad-hoc exploration |
Loki excels when you know what you're looking for (e.g., logs from service=X in namespace=Y). For arbitrary full-text searches across vast, unstructured logs, traditional systems might be more performant but at a significantly higher cost.
Zero-code auto-instrumentation using eBPF sidecars to feed traces and metrics into your LGTM pipeline.
Pipe k6 metrics directly into Mimir and visualize SLO breaches in Grafana dashboards during load tests.
Clean AI CLI logs, build cron expressions, decode JWTs, and calculate chmod permissions offline.

Automating blue-green and canary deployments on Kubernetes with Argo Rollouts, Prometheus analysis metrics, and automated instant rollbacks.
Read more
Design production Prometheus and Grafana alerts using multi-window multi-burn-rate PromQL rules for service-level objectives (SLOs) and error budgets.
Read more
Deep dive into eBPF observability in Kubernetes: eliminating Envoy sidecars, kernel probes, BPF ring buffers, and zero-code telemetry instrumentation.
Read more