Redis Memory Optimization: Internals, Data Structure Encoding, and Memory Profiling

Your Redis instance is a finely-tuned engine, but as your application scales, its memory consumption starts looking less like a sports car and more like a freight train. The cloud bill climbs, OOM-killer warnings flood your logs, and you're facing a classic engineering crossroads: throw more money at it with a larger instance, or embark on a complex sharding project. There is a third, more elegant path: a deep, surgical optimization of Redis's memory usage. This isn't about simple key eviction policies; it's about understanding how Redis actually stores your data at a low level and manipulating its behavior to achieve dramatic memory savings - often upwards of 70% - without sacrificing core performance.

The Anatomy of a Redis Key: More Than Just Your Data

When you SET mykey "hello", Redis doesn't just store the 5 bytes of "hello". Every key in Redis is associated with a fundamental metadata structure, and the value itself has its own overhead. Understanding this overhead is the first step toward minimizing it.

The redisObject Struct: The Root of All Keys

Every key in the main Redis dictionary (dict) doesn't point directly to your data (the string, the list, etc.). It points to a redisObject struct, the universal container for all data types. In C, it looks something like this:

// A simplified representation of the redisObject struct
typedef struct redisObject {
    unsigned type:4;        // Type of the object (String, List, Hash, etc.)
    unsigned encoding:4;    // Internal representation (raw, embstr, listpack, etc.)
    unsigned lru:24;        // LRU/LFU clock value for eviction
    int refcount;           // Reference count for shared objects
    void *ptr;              // Pointer to the actual data
} robj;

On a 64-bit system, this struct alone costs 4 + 4 + 4 + 8 = 20 bytes, but due to memory alignment, it typically occupies 24 bytes. This is a fixed, per-key overhead you cannot escape. Add to that the overhead from the main keyspace dictionary (dictEntry), which includes pointers to the key, the value (redisObject), and the next entry in a hash chain, adding another 24 bytes.

Takeaway: Every single key, no matter how small its value, has a baseline overhead of approximately 48 bytes before you even store one byte of data. Storing millions of tiny keys (e.g., SET user:123:online 1) is a recipe for memory bloat due to this fixed metadata tax.

String Encoding: SDS, embstr, and raw

Redis doesn't use standard C null-terminated strings. It uses a custom structure called a Simple Dynamic String (SDS). This structure is crucial for performance and safety, but it has memory implications.

An SDS header (sdshdr) is prepended to the actual string data:

// Simplified sdshdr8 for strings up to 2^8-1=255 bytes long
struct __attribute__ ((__packed__)) sdshdr8 {
    uint8_t len;     /* used */
    uint8_t alloc;   /* excluding the header and null terminator */
    unsigned char flags; /* 3 lsb of type, 5 lsb of unused bits */
    char buf[];      /* the actual string data */
};

The len tracks the string length (making STRLEN an O(1) operation), and alloc tracks the allocated memory, allowing for efficient appends without reallocating every time.

Redis has a critical optimization for how it stores the redisObject and the SDS string together:

1. OBJ_ENCODING_EMBSTR (Embedded String): For small strings (by default, 44 bytes or less), Redis allocates a single, contiguous block of memory for both the redisObject and the SDS string. This improves locality and reduces fragmentation, saving one malloc call and the associated metadata overhead (typically 8 bytes).

2. OBJ_ENCODING_RAW: For strings larger than 44 bytes, Redis performs two separate allocations: one for the redisObject and another for the SDS string (sdshdr + data). The redisObject's ptr field points to this second memory block.

Let's compare the memory for a 10-byte string like "locionic":

• embstr encoding:
  • redisObject + sdshdr8 + "locionic" + \0 = contiguous block
  • Total Size ≈ 24 (aligned robj size) + 10 (string) + 1 (null) ≈ ~35 bytes (allocator rounds up).
• raw encoding (if embstr didn't exist):
  • redisObject = 24 bytes
  • malloc overhead for ptr = 8 bytes
  • sdshdr8 + "locionic" + \0 = 1 + 1 + 1 + 10 + 1 = 14 bytes, rounded up to 16 by allocator.
  • Total Size = 24 + 8 + 16 = 48 bytes.

The embstr encoding provides a ~27% saving in this case. You can configure the embstr limit in newer Redis versions, but the default of 44 is chosen carefully to align with allocator block sizes.

Beyond Raw: Compact Data Structure Encodings

The most significant memory savings in Redis come from its use of specialized, memory-optimized encodings for Hashes, Lists, Sets, and Sorted Sets. When these collections are small, Redis avoids using memory-hungry data structures like hash tables and skiplists in favor of flat, contiguous memory layouts.

The Successor: Listpacks (Redis 7.0+)

For decades, ziplist was the workhorse of memory optimization. However, it had a critical flaw: cascading updates. An insertion or deletion in the middle of a large ziplist could require all subsequent entries to be shifted, leading to O(N²) complexity in the worst case.

Enter the listpack, the default encoding for small Hashes and Zsets since Redis 7.0. A listpack is a sequence of entries stored in a single contiguous block of memory, designed to eliminate the cascading update problem.

A listpack has a simple structure:

• A 6-byte header: 4 bytes for total bytes, 2 bytes for the number of elements.
• A sequence of entries.
• A 1-byte end-of-listpack marker (0xFF).

The magic is inside each entry. An entry stores:

1. Encoding Type & Data: The data itself, prefixed by bits indicating its type and length.
2. Entry-Total-Length: A variable-length encoding of the total size of this entry.

This structure allows for iterating forward or backward through the list. Crucially, since each entry knows its own total length, updating an entry in place (if it fits) or replacing it with a new one doesn't require any changes to subsequent entries. This solves the cascading update problem of ziplists.

The Transition: From Listpack to Hashtable

A Hash will be stored as a listpack as long as it meets two criteria:

1. The number of field-value pairs is less than hash-max-listpack-entries (default 512).
2. The size of the largest entry (field or value) is less than hash-max-listpack-value (default 64 bytes).

If either of these thresholds is crossed, Redis automatically converts the listpack into a standard hashtable (OBJ_ENCODING_HT). This conversion is a one-way street.

Let's see the impact with a Python script and the MEMORY USAGE command.

# populate_redis.py
import redis

# Connect to a local Redis instance
r = redis.Redis(decode_responses=True)

# Small hash, will be stored as a listpack
small_hash_key = "user:101:small"
r.delete(small_hash_key)
r.hset(small_hash_key, mapping={
    "name": "Bob Smith",
    "email": "bob.smith@example.com",
    "city": "New York",
    "status": "active"
})

# Large hash, will be forced into hashtable encoding by number of entries
large_hash_key = "user:102:large"
r.delete(large_hash_key)
# Default hash-max-listpack-entries is 512, so 513 will trigger conversion
large_hash_data = {f"field_{i}": f"value_{i}" for i in range(513)}
r.hset(large_hash_key, mapping=large_hash_data)

print(f"Profiling key: {small_hash_key}")
print(f"Profiling key: {large_hash_key}")

Now, let's run this and inspect the memory in redis-cli:

$ python populate_redis.py
Profiling key: user:101:small
Profiling key: user:102:large

$ redis-cli
127.0.0.1:6379> OBJECT ENCODING user:101:small
"listpack"
127.0.0.1:6379> MEMORY USAGE user:101:small
(integer) 166

# Now for the large one, which just barely crossed the threshold
127.0.0.1:6379> OBJECT ENCODING user:102:large
"hashtable"
127.0.0.1:6379> MEMORY USAGE user:102:large
(integer) 44104

The small hash with 4 fields uses only 166 bytes. The large hash, with 513 fields, consumes over 44 KB! This is not a linear increase. The jump is caused by the conversion to a full-fledged hash table, which involves creating dict and dictEntry structures, redisObjects for every field and value, and significant pointer overhead.

Special Cases: Intsets and Quicklists

Intsets (OBJ_ENCODING_INTSET): When a Set contains only 64-bit signed integers, Redis uses a highly compact intset encoding. This is essentially a sorted array of integers. It's incredibly memory-efficient. However, the moment you add a non-integer element to the set, it's permanently converted to a standard hash table.

• Threshold: set-max-intset-entries (default 512).

Quicklists (for Lists): A Redis List isn't a simple linked list. It's a quicklist: a linked list of listpacks. Each node in the quicklist (quicklistNode) contains a listpack that can hold multiple list elements. This is a brilliant compromise. It avoids the overhead of a linked list where every single element is a separate memory allocation, but it also avoids having one massive contiguous block for the entire list.

You can tune this behavior with two key settings:

• list-max-listpack-size: Controls the maximum size of each internal listpack. A negative value sets a byte limit (e.g., -2 for 8KB per node). This is the recommended way to configure it.
• list-compress-depth: A fascinating optimization. This allows you to compress listpack nodes at the head and tail of the quicklist that are not accessed frequently. 0 disables compression. 1 means the outermost nodes (head and tail) are uncompressed, the next level in is compressed, and so on. This can save significant memory for very long lists where you typically only access the ends (e.g., using Redis as a queue).

From Theory to Practice: Profiling and Tuning

Armed with this internal knowledge, we can now take concrete steps to optimize our instance.

Step 1: Profile with the MEMORY Command

The MEMORY command is your primary tool for investigating memory issues.

• MEMORY USAGE <key>: As shown above, gives the total bytes allocated for a specific key. This is your go-to for checking if a key is using a compact encoding.
• MEMORY STATS: Provides a detailed breakdown of the server's memory usage, including keys.count, keys.bytes-per-key, and detailed breakdowns by data type.
• MEMORY DOCTOR: A built-in diagnostic tool. Redis will analyze its state and report potential issues, such as high fragmentation or inefficient data structures. It's a great starting point.

A common task is to find your largest or most problematic keys. You can do this with a simple script:

# WARNING: The --bigkeys scan can cause a small latency spike.
# Use it with caution on a production primary.
# It is safer to run on a replica.

redis-cli --bigkeys

# --- Sample Output ---
# ...
# [00.00%] Biggest zset found so far: 'my_leaderboard' with 100002 members
# [00.00%] Biggest hash found so far: 'user:session:fat' with 513 fields
# ...
#
# Summary:
# Sampled 123456 keys in the keyspace.
#
# Biggest zset: 'my_leaderboard' (100002 members)
# Biggest hash: 'user:session:fat' (513 fields)
#
# 98765 strings with 1.23 GB
# 12345 lists with 50.44 MB
# 5432 hashes with 2.10 GB
# 1234 zsets with 450.12 MB
# 1 set with 12 bytes

This immediately points you to keys that have likely broken out of their compact encodings.

Step 2: Tune Encoding Thresholds

If your profiling reveals a common data access pattern, you can tune the encoding thresholds in your redis.conf or via CONFIG SET.

Scenario: You store user session data in hashes. Most sessions have around 60-70 fields, but occasionally some cross the 64-byte hash-max-listpack-value limit for a single field, forcing the entire hash into the inefficient hashtable encoding.

Action: Analyze the data. If the large fields are rare and not performance-critical, you could consider increasing the value limit.

# In your redis.conf

# Default: 64. Let's increase it to allow slightly larger values in our hashes.
hash-max-listpack-value 128

# Default: 512. If your hashes are consistently just over this limit,
# and write performance isn't the absolute top priority for them,
# a modest increase can save a lot of memory.
hash-max-listpack-entries 1024

Warning: There is a CPU/memory trade-off. Larger listpacks are more memory-efficient, but finding an element within them is an O(N) operation. For a hash with 1000 entries, finding a key might require scanning all 1000 entries. A hashtable provides O(1) access. Tune these values based on your specific workload's read/write patterns and latency requirements.

Step 3: Taming Memory Fragmentation

Memory fragmentation is a silent memory killer in long-running Redis instances. Redis allocates and frees memory constantly. The memory allocator (jemalloc by default) may not be able to release freed memory back to the operating system, leading to a discrepancy between what Redis reports (used_memory) and what the OS sees the process using (RSS - Resident Set Size).

You can check your fragmentation ratio via INFO MEMORY:

127.0.0.1:6379> INFO MEMORY
...
used_memory:104857600       # 100 MB used by Redis data
used_memory_rss:136314880    # 130 MB from OS perspective
...
mem_fragmentation_ratio:1.30
...

A ratio above 1.5 is high and indicates significant fragmentation. A ratio below 1.0 means Redis is swapping, which is a critical problem.

Redis 4.0+ introduced Active Defragmentation, a feature that allows Redis to combat fragmentation automatically in the background.

# In your redis.conf - safe starting values
activedefrag yes

# Don't start defragging unless fragmentation is at least 100MB
active-defrag-ignore-bytes 100mb

# Start defragging when fragmentation ratio reaches 1.1 (10%)
active-defrag-threshold-lower 10

# Stop defragging when the CPU usage for defrag hits 5%
# to minimize impact on the main thread.
active-defrag-cycle-max 5

When enabled, Redis will use spare CPU cycles to scan memory, identify fragmented values, and move them to contiguous memory blocks, allowing the allocator to reclaim the now-free fragmented space. For any production system with a non-trivial write load, enabling active defrag is highly recommended.

Final Takeaways and Action Plan

Optimizing Redis memory is not a dark art; it's an engineering discipline grounded in understanding its internal data structures. By moving from theory to practice, you can reclaim vast amounts of RAM and run a more stable, cost-effective system.

1. Audit Your Data Model: Are you using many small keys where a single hash would do? Grouping related data is your biggest potential win.
2. Profile Your Keys: Use MEMORY USAGE and --bigkeys to identify keys that have inefficiently converted to hashtable or skiplist encodings.
3. Tune Your redis.conf: Adjust the *-max-listpack-* directives based on your application's data patterns. A small increase can prevent premature conversion and save gigabytes.
4. Embrace Active Defragmentation: Enable activedefrag on any long-running, write-heavy Redis instance. It is the single best tool for managing the health of your server's memory over time.
5. Monitor Fragmentation: Keep an eye on the mem_fragmentation_ratio. If it stays stubbornly high even with activedefrag, it may point to a deeper issue with allocation patterns or a need for a planned restart during a maintenance window.

By applying these principles, you're not just cutting costs; you're building a more robust and scalable architecture.

Further Reading

• Redis Documentation on Memory Optimization
• Salvatore Sanfilippo's (antirez) post on SDS
• Understanding Redis quicklist (Note: The linked article discusses RocksDB's use of a similar concept, but the Redis implementation details are in the Redis source code).
• Jemalloc Documentation - For the truly curious about the underlying memory allocator.

Test Your Knowledge