
This guide details the architectural considerations, configuration, and deployment strategies for the OpenTelemetry Collector within a Kubernetes environment, focusing on scalability, advanced sampling, and direct export to ClickHouse. The objective is to provide a robust, production-ready observability pipeline with zero data loss guarantees.
Deploying the OpenTelemetry Collector in Kubernetes necessitates a clear architectural strategy. The primary models are Agent (DaemonSet), Gateway (Deployment), or a Hybrid approach. Each model presents distinct trade-offs in terms of resource utilization, latency, and processing capabilities.
The Agent architecture deploys an OpenTelemetry Collector instance on every Kubernetes node. This is typically achieved using a DaemonSet, ensuring a collector pod runs on each node. Agents are primarily responsible for local data collection and initial processing.
Characteristics:
Advantages:
Disadvantages:
Use Cases:
Example: OpenTelemetry Collector Agent DaemonSet
This example demonstrates a basic DaemonSet for an OTel Collector Agent, configured to receive OTLP and Prometheus metrics, and forward them.
apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
name: otel-agent
namespace: observability
spec:
mode: daemonset
image: otel/opentelemetry-collector-contrib:0.90.1
hostNetwork: true # Required for host-level scraping
serviceAccount: otel-collector
resources:
requests:
memory: "128Mi"
cpu: "100m"
limits:
memory: "256Mi"
cpu: "200m"
config: |
receivers:
otlp:
protocols:
grpc:
http:
prometheus:
config:
scrape_configs:
- job_name: 'kubernetes-nodes'
static_configs:
- targets: ['localhost:9100'] # Node Exporter
- job_name: 'kubernetes-pods'
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
action: keep
regex: true
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
action: replace
target_label: __metrics_path__
regex: (.+)
- source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
action: replace
regex: ([^:]+)(?::\d+)?;(\d+)
replacement: $1:$2
target_label: __address__
processors:
batch:
send_batch_size: 10000
timeout: 10s
memory_limiter:
limit_mib: 150
spike_limit_mib: 50
check_interval: 1s
resourcedetection:
detectors: [env, system, kubernetes]
timeout: 2s
override: true
exporters:
otlp:
endpoint: "otel-gateway.observability.svc.cluster.local:4317" # Forward to Gateway
tls:
insecure: true
service:
pipelines:
traces:
receivers: [otlp]
processors: [resourcedetection, batch, memory_limiter]
exporters: [otlp]
metrics:
receivers: [otlp, prometheus]
processors: [resourcedetection, batch, memory_limiter]
exporters: [otlp]
The Gateway architecture employs a centralized OpenTelemetry Collector deployment, typically as a Kubernetes Deployment, scaled horizontally. These collectors receive data from Agents or directly from applications, perform advanced processing, and export to various backends.
Characteristics:
Advantages:
Disadvantages:
Use Cases:
Example: OpenTelemetry Collector Gateway Deployment
This example shows a Deployment for an OTel Collector Gateway, configured to receive OTLP, process, and export.
apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
name: otel-gateway
namespace: observability
spec:
mode: deployment
image: otel/opentelemetry-collector-contrib:0.90.1
replicas: 3 # Scaled for high availability and throughput
serviceAccount: otel-collector
ports:
- name: otlp-grpc
port: 4317
targetPort: 4317
protocol: TCP
- name: otlp-http
port: 4318
targetPort: 4318
protocol: TCP
- name: metrics
port: 8888
targetPort: 8888
protocol: TCP
resources:
requests:
memory: "1Gi"
cpu: "500m"
limits:
memory: "2Gi"
cpu: "1000m"
config: |
receivers:
otlp:
protocols:
grpc:
http:
processors:
batch:
send_batch_size: 10000
timeout: 10s
memory_limiter:
limit_mib: 1500
spike_limit_mib: 500
check_interval: 1s
# Tail sampling configuration will be added here later
exporters:
# ClickHouse exporter will be added here later
logging:
verbosity: detailed
service:
pipelines:
traces:
receivers: [otlp]
processors: [batch, memory_limiter] # Tail sampling will be inserted here
exporters: [logging] # ClickHouse exporter will replace/augment this
metrics:
receivers: [otlp]
processors: [batch, memory_limiter]
exporters: [logging] # ClickHouse exporter will replace/augment this
The Hybrid architecture combines the strengths of both Agent and Gateway models. Agents collect data locally and forward it to Gateways, which then perform advanced processing and export. This is the recommended approach for most production environments.
Characteristics:
Advantages:
Disadvantages:
Example: Agent forwarding to Gateway (Agent config snippet)
The Agent configuration from Section 1.1 already demonstrates forwarding to a Gateway:
exporters:
otlp:
endpoint: "otel-gateway.observability.svc.cluster.local:4317" # Forward to Gateway
tls:
insecure: true
| Feature | Agent (DaemonSet) | Gateway (Deployment) | Hybrid (Agent + Gateway) |
|---|---|---|---|
| Deployment Model | DaemonSet (per node) | Deployment (centralized, scaled) | DaemonSet (Agents) + Deployment (Gateways) |
| Resource Utilization | Per-node overhead, distributed | Centralized, potentially higher per-instance | Higher overall, distributed collection, centralized processing |
| Latency (App->Collector) | Negligible (local IPC/loopback) | 5-10ms (network hop) | Negligible (App->Agent), then 5-10ms (Agent->Gateway) |
| Sampling Capability | Head-based only, no global context | Tail-based, probabilistic, global context | Head-based (Agent), Tail-based (Gateway) |
| Resilience | Node-level isolation, high local resilience | Scaled for resilience, but central point of failure if not scaled | High resilience at both layers |
| Complexity | Low | Medium | High |
| Best Use Case | Host metrics, logs, initial trace collection | Advanced processing, global sampling, fan-out | Most production environments, comprehensive observability |
| Data Loss Risk | Lower for local collection, higher for export if no buffering | Higher if not scaled or no buffering | Lower with proper buffering at both layers |
Effective OpenTelemetry Collector operation relies on a well-structured configuration. This section covers fundamental processors and exporters crucial for production deployments.
The OpenTelemetry Collector configuration is declarative and structured around four main components:
batch, memory_limiter, tail_sampling, resourcedetection).The batch processor is critical for optimizing network utilization and reducing the load on backend systems. It aggregates telemetry data into larger batches before exporting.
Benefits:
Configuration Parameters:
send_batch_size: The maximum number of telemetry items (spans, metric data points, log records) to send in a single batch.timeout: The maximum duration to wait before sending a batch, even if send_batch_size is not reached.send_batch_max_size: The absolute maximum size of a batch, overriding send_batch_size if necessary to prevent excessively large batches.Example: batch processor configuration
processors:
batch:
send_batch_size: 10000 # Max items per batch
timeout: 10s # Max wait time before sending
send_batch_max_size: 12000 # Absolute max items, useful for high-volume bursts
The memory_limiter processor is essential for preventing the OpenTelemetry Collector from consuming excessive memory and being OOMKilled by Kubernetes. It monitors memory usage and, if limits are approached, drops data to prevent crashes.
Benefits:
Configuration Parameters:
limit_mib: The maximum amount of memory (in MiB) the collector is allowed to use. When this limit is reached, data is dropped. This should be less than the container's Kubernetes memory limit.spike_limit_mib: The maximum amount of memory (in MiB) the collector can temporarily exceed limit_mib by. This helps handle sudden spikes without immediate data drops.check_interval: How frequently (duration) the memory usage is checked.Example: memory_limiter processor configuration
processors:
memory_limiter:
limit_mib: 1500 # Max memory in MiB before dropping data
spike_limit_mib: 500 # Allowed spike above limit_mib
check_interval: 1s # How often to check memory usage
Note: limit_mib should always be set lower than the Kubernetes container's limits.memory to allow the collector to react before Kubernetes OOMKills it. A good rule of thumb is limit_mib = limits.memory * 0.75.
When using a Hybrid architecture, Agents need to distribute traces across multiple Gateway instances for high availability and load distribution. The loadbalancing exporter facilitates this.
Benefits:
Configuration Parameters:
protocol: The OTLP protocol to use (grpc or http).resolver: Defines how target endpoints are discovered.
dns: Uses DNS SRV records or A records to discover endpoints. Recommended for Kubernetes services.static: A fixed list of endpoints. Less dynamic.routing_key: Determines how traces are routed. traceID is the default and recommended to ensure all spans of a single trace go to the same Gateway for proper processing (e.g., tail sampling).Example: Agent loadbalancing exporter configuration
exporters:
otlp/loadbalancer:
endpoint: "otel-gateway.observability.svc.cluster.local:4317" # Kubernetes Service DNS
tls:
insecure: true
loadbalancing:
resolver:
dns:
hostname: otel-gateway.observability.svc.cluster.local # Service DNS name
port: 4317
routing_key: traceID # Ensures all spans of a trace go to the same gateway
service:
pipelines:
traces:
receivers: [otlp]
processors: [resourcedetection, batch, memory_limiter]
exporters: [otlp/loadbalancer] # Use the loadbalancing exporter
Sampling is crucial for managing the volume of telemetry data, especially traces. While head-based sampling (deciding at the start of a trace) is simple, it often discards valuable context. Tail-based sampling addresses this by making sampling decisions after a trace has completed.
Tail-based sampling allows for intelligent sampling decisions based on the entire trace context. This means you can reliably capture:
Benefits:
Trade-offs:
tail_sampling Processor ConfigurationThe tail_sampling processor is configured within the Gateway Collector. It holds traces for a specified duration (decision_wait) to gather all spans before applying sampling policies.
Key Parameters:
decision_wait: The maximum duration to wait for all spans of a trace to arrive before making a sampling decision. This is critical for ensuring trace completeness.num_traces: The maximum number of traces to hold in memory concurrently. This directly impacts memory usage.expected_new_traces_per_sec: An estimate of the incoming trace rate, used for internal sizing.Policy Types:
The tail_sampling processor supports various policies, which can be combined using and or or composite policies:
always_sample: Always samples the trace.drop_new: Drops new traces if num_traces is exceeded.probabilistic: Samples traces based on a configured probability.rate_limiting: Samples traces at a maximum rate per second.status_code: Samples traces based on HTTP status codes (e.g., ERROR, UNSET).latency: Samples traces exceeding a specified latency threshold.attribute: Samples traces based on specific span or resource attributes.composite: Combines multiple policies using and or or logic.Example: tail_sampling processor with composite policy
This configuration samples all traces that contain an error OR have a latency greater than 500ms.
processors:
# ... other processors like batch, memory_limiter
tail_sampling:
decision_wait: 10s # Wait up to 10 seconds for all spans of a trace
num_traces: 100000 # Max traces to hold in memory
expected_new_traces_per_sec: 1000 # Expected incoming rate
policies:
- name: error-or-slow-trace
type: composite
composite:
or_policies:
- name: error-policy
type: status_code
status_code:
status_codes: [ERROR]
- name: slow-trace-policy
type: latency
latency:
threshold_ms: 500 # Sample traces > 500ms
on_failure: always_sample # If composite policy fails, always sample
Integration into Gateway Pipeline:
service:
pipelines:
traces:
receivers: [otlp]
processors: [batch, memory_limiter, tail_sampling] # Insert tail_sampling here
exporters: [logging] # Exporters will follow
ClickHouse is an analytical database optimized for high-throughput ingestion and fast query execution, making it an excellent choice for storing OpenTelemetry traces and metrics. The clickhouse exporter allows direct integration.
The clickhouse exporter sends traces, metrics, and logs directly to ClickHouse tables. It supports various configurations for table schemas, TTLs, and compression.
Traces are typically stored in a table designed to efficiently query by trace_id, service_name, operation_name, and time ranges.
Schema Considerations for ClickHouse Traces: A common schema for OpenTelemetry traces in ClickHouse includes:
trace_id (FixedString(16))span_id (FixedString(8))parent_span_id (FixedString(8))service_name (String)operation_name (String)start_time_unix_nano (UInt64)duration_nano (Int64)kind (Int8)status_code (Int8)status_message (String)attributes (Map(String, String)) - for span attributesresource_attributes (Map(String, String)) - for resource attributesevents.timestamp_unix_nano (Array(UInt64))events.name (Array(String))events.attributes (Array(Map(String, String)))links.trace_id (Array(FixedString(16)))links.span_id (Array(FixedString(8)))links.attributes (Array(Map(String, String)))Example: ClickHouse exporter configuration for traces
exporters:
clickhouse/traces:
dsn: "tcp://clickhouse-cluster.observability.svc.cluster.local:9000?database=otel"
database: otel
traces_table: otel_traces
ttl: 7d # Data retention for 7 days
compression: lz4 # Use LZ4 compression
# Optional: Define specific schema mappings if default is not sufficient
# trace_id_field: trace_id
# span_id_field: span_id
# ...
ClickHouse DDL for otel_traces table:
CREATE TABLE otel_traces (
Timestamp DateTime64(9) CODEC(Delta, ZSTD(1)),
TraceId FixedString(16) CODEC(ZSTD(1)),
SpanId FixedString(8) CODEC(ZSTD(1)),
ParentSpanId FixedString(8) CODEC(ZSTD(1)),
TraceState String CODEC(ZSTD(1)),
SpanName String CODEC(ZSTD(1)),
SpanKind Int8 CODEC(ZSTD(1)),
ServiceName LowCardinality(String) CODEC(ZSTD(1)),
ResourceSchemaUrl String CODEC(ZSTD(1)),
ResourceAttributes Map(LowCardinality(String), String) CODEC(ZSTD(1)),
ScopeSchemaUrl String CODEC(ZSTD(1)),
ScopeName String CODEC(ZSTD(1)),
ScopeVersion String CODEC(ZSTD(1)),
SpanAttributes Map(LowCardinality(String), String) CODEC(ZSTD(1)),
DurationNanos UInt64 CODEC(ZSTD(1)),
StatusCode Int8 CODEC(ZSTD(1)),
StatusMessage String CODEC(ZSTD(1)),
Events Nested (
Timestamp DateTime64(9),
Name String,
Attributes Map(LowCardinality(String), String)
) CODEC(ZSTD(1)),
Links Nested (
TraceId FixedString(16),
SpanId FixedString(8),
TraceState String,
Attributes Map(LowCardinality(String), String)
) CODEC(ZSTD(1))
) ENGINE = MergeTree()
PARTITION BY toYYYYMMDD(Timestamp)
ORDER BY (ServiceName, SpanName, Timestamp, TraceId)
TTL Timestamp + INTERVAL 7 DAY
SETTINGS index_granularity = 8192, ttl_only_drop_parts = 1;
Metrics are typically stored in a table optimized for time-series
A large-scale Kubernetes cluster experiences intermittent network latency spikes between nodes. An engineering team is deploying OpenTelemetry Collectors to gather host-level metrics and application traces. They prioritize minimizing the impact of telemetry collection on application performance and ensuring data is collected even if a central gateway becomes temporarily unavailable. Which OpenTelemetry Collector architecture would be most suitable for their initial deployment strategy?
Clean AI CLI logs, build cron expressions, decode JWTs, and calculate chmod permissions offline.

Deploying OpenTelemetry Collector with eBPF auto-instrumentation in Kubernetes: zero-code HTTP/gRPC tracing and Prometheus pipeline export.
Read more
Ship unified observability with one OTel Collector: route logs to Loki, metrics to Mimir, and traces to Tempo. Includes Helm charts, multi-tenant config, cross-signal correlation, and S3 cost tips.
Read more
Ingesting 50,000 trace spans/sec with OpenTelemetry Collector, Kafka buffering, and ClickHouse columnar storage without losing spans or blowing budgets.
Read more