
Apache Kafka's traditional architecture tightly couples compute and storage. Brokers manage local log segments, typically backed by high-performance block storage like AWS EBS or GCP Persistent Disk. While this design offers unparalleled throughput and low-latency access for recent data, it becomes economically unsustainable for long-term data retention, especially as data volumes scale into petabytes. The cost of provisioned IOPS and storage capacity on block devices for historical, infrequently accessed data can quickly dominate infrastructure budgets.
Kafka Tiered Storage, introduced as KIP-405, fundamentally re-architects Kafka's storage layer. It enables the seamless offloading of older, less frequently accessed log segments from local broker storage to cost-effective, highly durable object storage solutions like AWS S3 and Google Cloud Storage (GCS). This separation of compute and storage allows for significant cost reductions, enhanced scalability, and simplified operational management without compromising Kafka's core guarantees.
KIP-405 introduces a two-tiered storage model: a local tier on the Kafka brokers and a remote tier on object storage. The primary objective is to leverage the cost-effectiveness and virtually infinite scalability of cloud object storage for historical data while retaining the performance benefits of local storage for hot data.
The tiered storage architecture fundamentally alters how Kafka manages its log segments. Understanding the data flow and component interactions is critical for effective deployment and optimization.
log.segment.bytes) or time (log.segment.ms) limit, it is rolled over, becoming an immutable, closed segment.local.retention.bytes or local.retention.ms to determine which segments to keep locally. Segments older than this policy are marked for offloading.RemoteLogManager asynchronously uploads these marked segments to the configured object storage bucket (S3 or GCS). This process is non-blocking to broker operations.remote.log.storage.enable: Global broker setting to enable/disable tiered storage. Must be true.remote.log.storage.manager.impl.class: Specifies the implementation class for the remote storage manager.
org.apache.kafka.server.log.remote.metadata.storage.S3RemoteLogMetadataManagerorg.apache.kafka.server.log.remote.metadata.storage.GcsRemoteLogMetadataManagerremote.log.storage.manager.impl.prefix: A prefix for all remote storage manager related configurations. E.g., remote.log.storage.manager.impl.aws.region.local.retention.bytes: The maximum number of bytes to retain on local disk per partition. When this limit is exceeded, older segments are offloaded.local.retention.ms: The maximum time (in milliseconds) to retain segments on local disk per partition.log.segment.bytes: The maximum size of a log segment file. Smaller segments mean more frequent rollovers and uploads, potentially increasing object storage API calls.log.segment.ms: The maximum time before a log segment is rolled over.Implementing Kafka Tiered Storage requires careful configuration at both the broker and topic levels. We will detail the setup for AWS S3 and Google Cloud Storage.
s3:PutObject, s3:GetObject, s3:DeleteObject, s3:ListBucket, s3:GetBucketLocation on the target S3 bucket.Storage Object Admin or Storage Object Creator and Storage Object Viewer roles on the target GCS bucket.server.properties)The following parameters are added to server.properties on each Kafka broker.
# Enable remote storage
remote.log.storage.enable=true
# Set the local retention policy.
# Example: Retain 10GB per partition locally.
# This value should be carefully chosen based on hot data access patterns.
local.retention.bytes=10737418240 # 10 GB
# Alternatively, or in addition to bytes, set time-based local retention.
# Example: Retain 24 hours of data locally.
# local.retention.ms=86400000 # 24 hours
# Configure the segment size and time for optimal offloading.
# Smaller segments mean more frequent uploads but faster local deletion.
log.segment.bytes=1073741824 # 1 GB
log.segment.ms=604800000 # 7 days (default)
# S3 Remote Log Storage Manager implementation class
remote.log.storage.manager.impl.class=org.apache.kafka.server.log.remote.metadata.storage.S3RemoteLogMetadataManager
# S3 bucket name for remote storage
remote.log.storage.manager.impl.aws.bucket.name=your-kafka-tiered-storage-s3-bucket
# AWS region of the S3 bucket
remote.log.storage.manager.impl.aws.region=us-east-1
# (Optional) AWS endpoint for S3, useful for S3-compatible storage or private endpoints
# remote.log.storage.manager.impl.aws.endpoint=https://s3.us-east-1.amazonaws.com
# (Optional) AWS credentials provider. Default is DefaultAWSCredentialsProviderChain.
# remote.log.storage.manager.impl.aws.credentials.provider.class=com.amazonaws.auth.DefaultAWSCredentialsProviderChain
# (Optional) Number of threads for S3 uploads/downloads
# remote.log.storage.manager.impl.aws.num.client.threads=10
# GCS Remote Log Storage Manager implementation class
remote.log.storage.manager.impl.class=org.apache.kafka.server.log.remote.metadata.storage.GcsRemoteLogMetadataManager
# GCS bucket name for remote storage
remote.log.storage.manager.impl.gcp.bucket.name=your-kafka-tiered-storage-gcs-bucket
# GCP Project ID
remote.log.storage.manager.impl.gcp.project.id=your-gcp-project-id
# Path to the service account key file (JSON).
# Ensure this file is accessible by the Kafka broker process.
# Alternatively, if running on GCP, use default credentials via instance service account.
# remote.log.storage.manager.impl.gcp.credentials.file=/path/to/your/gcp-service-account-key.json
# (Optional) Number of threads for GCS uploads/downloads
# remote.log.storage.manager.impl.gcp.num.client.threads=10
Tiered storage can be enabled and configured at the topic level, overriding broker-level defaults.
# Create a new topic with remote storage enabled and specific local retention
kafka-topics.sh --create --topic my-tiered-topic \
--bootstrap-server localhost:9092 \
--partitions 3 --replication-factor 3 \
--config remote.storage.enable=true \
--config local.retention.bytes=5368709120 # 5 GB local retention for this topic
# Modify an existing topic to enable remote storage
kafka-configs.sh --bootstrap-server localhost:9092 --entity-type topics --entity-name my-existing-topic \
--alter --add-config remote.storage.enable=true
# Modify local retention for an existing tiered topic
kafka-configs.sh --bootstrap-server localhost:9092 --entity-type topics --entity-name my-existing-topic \
--alter --add-config local.retention.ms=172800000 # 48 hours local retention
Strimzi simplifies running Kafka on Kubernetes. To enable tiered storage, you modify the Kafka custom resource.
Kafka CR ExampleapiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
name: my-kafka-cluster
spec:
kafka:
version: 3.6.0 # Ensure Kafka version supports KIP-405
replicas: 3
listeners:
- name: plain
port: 9092
type: internal
tls: false
- name: tls
port: 9093
type: internal
tls: true
storage:
type: jbod
volumes:
- id: 0
type: persistent-claim
size: 100Gi # Smaller local storage needed due to tiered storage
deleteClaim: false
config:
# Common Tiered Storage Settings
remote.log.storage.enable: "true"
local.retention.bytes: "10737418240" # 10 GB
log.segment.bytes: "1073741824" # 1 GB
# AWS S3 Specific Configuration
remote.log.storage.manager.impl.class: "org.apache.kafka.server.log.remote.metadata.storage.S3RemoteLogMetadataManager"
remote.log.storage.manager.impl.aws.bucket.name: "your-kafka-tiered-storage-s3-bucket"
remote.log.storage.manager.impl.aws.region: "us-east-1"
# For AWS, ensure the EC2 instance profile or Kubernetes service account has the necessary IAM role.
# Strimzi can inject environment variables for AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY if needed,
# but IAM roles are preferred.
# OR Google Cloud Storage Specific Configuration
# remote.log.storage.manager.impl.class: "org.apache.kafka.server.log.remote.metadata.storage.GcsRemoteLogMetadataManager"
# remote.log.storage.manager.impl.gcp.bucket.name: "your-kafka-tiered-storage-gcs-bucket"
# remote.log.storage.manager.impl.gcp.project.id: "your-gcp-project-id"
# For GCP, use Workload Identity to bind a Kubernetes Service Account to a GCP Service Account.
# The GCP Service Account should have Storage Object Admin permissions.
# Strimzi will automatically pick up credentials if the pod's service account is configured correctly.
# Example of how to configure a KafkaTopic with remote storage enabled
entityOperator:
topicOperator: {}
userOperator: {}
---
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
name: my-tiered-topic
labels:
strimzi.io/cluster: my-kafka-cluster
spec:
partitions: 3
replicas: 3
config:
remote.storage.enable: "true"
local.retention.bytes: "5368709120" # 5 GB local retention for this topic
s3:PutObject, s3:GetObject, s3:DeleteObject, s3:ListBucket, s3:GetBucketLocation permissions to the target S3 bucket.Storage Object Admin or equivalent permissions on the GCS bucket.The primary driver for Kafka Tiered Storage is cost reduction. Let's quantify the potential savings with a realistic scenario.
Scenario:
us-east-1 for AWS, us-east4 for GCP.For 1 year of retention, the total storage required is 30 TB/day * 365 days = 10,950 TB (approx 11 PB). This is an extreme case for traditional Kafka, often leading to shorter retention. Let's assume a more common 30-day retention for traditional Kafka due to cost constraints.
gp3 (cost-effective, balanced performance)gp3 Cost: $0.08/GB-month0.08/GB-month = **72,000/month**This calculation only covers storage. It does not include EC2 instance costs, network transfer, or IOPS. The high storage cost often forces organizations to drastically reduce retention.
With tiered storage, we can retain 30 days locally and 1 year remotely.
Local Storage (30 days):
local.retention.bytes is per partition. Let's assume 100GiB per broker for active data.gp3 Cost: 1,000 GB * 0.08/GB-month = **80/month**Remote Storage (1 year):
0.023/GB for first 50TB, 0.022/GB for next 450TB, etc. (average $0.022/GB-month for this scale)0.022/GB-month = **80,300/month**0.020/GB for first 1PB, etc. (average 0.020/GB-month for this scale)0.020/GB-month = **73,000/month**Object Storage API Calls:
0.005 per 1,000 requests. 10,000 * 30 days = 300,000 PUTs. Cost: 0.005 * 300 = $1.5/month. (Negligible)0.0004 per 1,000 requests. 0.0004 * 365 = $0.146/month. (Negligible)Total Tiered Storage Cost (AWS S3): 80 (Local EBS) + 80,300 (S3 Storage) + ~2 (API Calls) = **~80,382/month**
Total Tiered Storage Cost (GCS): 80 (Local EBS) + 73,000 (GCS Storage) + ~2 (API Calls) = **~73,082/month**
This comparison is tricky because traditional Kafka often cannot afford 1 year of retention. If we compare 30 days of retention on EBS vs. 1 year of retention with Tiered Storage:
This doesn't show savings, it shows increased cost for significantly increased retention. The true saving is in the cost per GB-month for long-term data.
Let's re-evaluate the scenario: What if we had to store 1 year of data on EBS?
gp3 Cost: 10,950,000 GB * 0.08/GB-month = **876,000/month**Now, compare this to Tiered Storage for 1 year:
Savings Calculation (AWS S3 vs. EBS for 1 year retention):
(876,000 - 80,382) / $876,000 = 0.908 = 90.8% savings
Savings Calculation (GCS vs. EBS for 1 year retention):
(876,000 - 73,082) / $876,000 = 0.916 = 91.6% savings
This demonstrates that for long-term retention, tiered storage offers over 90% savings on storage costs compared to an all-EBS solution. Even if you only needed 3 months of retention on EBS, the costs would be 3x higher than the 30-day EBS + 1-year S3/GCS tiered approach.
Furthermore, the compute costs for brokers can be reduced. With smaller local disk requirements, you might be able to use smaller EC2 instance types or reduce the number of brokers, further contributing to savings.
While cost savings are substantial, it's crucial to understand the performance implications of introducing an object storage layer.
local.retention window, performance remains identical to traditional Kafka. Latency is typically sub-millisecond.| Feature | Traditional Kafka (EBS/Persistent Disk) | Kafka Tiered Storage (Local EBS + Remote S3/GCS)
A Kafka cluster utilizing Tiered Storage (KIP-405) experiences a sudden, sustained spike in consumer requests for historical data (e.g., 3 months old) that has been offloaded to AWS S3. Which of the following is the MOST likely immediate operational impact?
Clean AI CLI logs, build cron expressions, decode JWTs, and calculate chmod permissions offline.

Ingesting 50,000 trace spans/sec with OpenTelemetry Collector, Kafka buffering, and ClickHouse columnar storage without losing spans or blowing budgets.
Read more
A battle-tested production guide to Google Cloud Platform. Learn why Cloud Run beats GKE for 90% of workloads, how to leverage BigQuery and Secret Manager, and the 5 hidden cost traps that drain cloud budgets.
Read more
Apache Iceberg vs Delta Lake benchmarked: ACID transaction isolation, schema evolution, hidden partitioning, and engine-agnostic query speeds on S3.
Read more