
ScyllaDB is a high-performance, NoSQL distributed database compatible with Apache Cassandra and Amazon DynamoDB APIs. Its architectural distinctiveness stems from its underlying C++ asynchronous engine, Seastar. This document dissects ScyllaDB's core architectural tenets: thread-per-core execution, shared-nothing design, NUMA awareness, and user-space I/O scheduling, elucidating how these principles collectively deliver predictable sub-millisecond P99 latencies even under extreme loads.
Traditional database systems, including Apache Cassandra, often rely on a multi-threaded architecture where a pool of threads processes requests. These threads contend for shared resources (locks, caches) and are subject to operating system scheduler interventions, leading to context switching overheads and unpredictable latency spikes. Furthermore, JVM-based systems like Cassandra introduce garbage collection (GC) pauses, which can significantly impact P99 latency.
Seastar, the asynchronous C++ framework powering ScyllaDB, fundamentally re-architects this model. It adopts a "shared-nothing, thread-per-core" execution paradigm.
In Seastar, each CPU core is assigned a dedicated Seastar thread (also known as a "shard" or "reactor"). This thread is pinned to its core and runs an event loop. All data and logic processed by that core are local to it. There is no shared memory or shared data structures between cores that require locks or atomic operations for concurrent access.
This design eliminates:
The "shared-nothing" principle extends beyond CPU cores to other resources. Each Seastar thread manages its own:
io_uring.When a request arrives, it's typically hashed to a specific core based on a partition key. That core then handles the request end-to-end, from network ingress to disk I/O and back, without involving other cores unless data needs to be explicitly transferred (e.g., for cross-shard queries, which are handled via message passing).
Seastar employs a future-based asynchronous programming model. Operations that would traditionally block (e.g., disk I/O, network I/O) return a future<T> object immediately. The Seastar event loop then schedules other ready tasks while the I/O operation completes. Once the I/O is done, the corresponding future is "fulfilled," and its continuation (a callback or chained operation) is scheduled for execution on the same core.
This cooperative multitasking within a single thread-per-core avoids the overhead of OS context switches while still allowing high concurrency.
// Example: Seastar asynchronous I/O
#include <seastar/core/app-template.hh>
#include <seastar/core/future.hh>
#include <seastar/core/file.hh>
#include <seastar/core/reactor.hh>
#include <seastar/core/thread.hh> // For seastar::thread
// Function to write data asynchronously to a file
seastar::future<> write_to_file(const seastar::sstring& filename, const seastar::sstring& data) {
// Open the file asynchronously. O_CREAT | O_TRUNC | O_WRONLY are standard flags.
// 0644 is file permissions.
return seastar::open_file_dma(filename, seastar::open_flags::rw_create | seastar::open_flags::truncate).then([data](seastar::file f) {
// Allocate a DMA-aligned buffer for efficient I/O
auto buffer = seastar::temporary_buffer<char>::aligned(4096, data.size());
std::copy(data.begin(), data.end(), buffer.begin());
// Write the buffer to the file asynchronously
return f.dma_write(buffer.get(), 0, buffer.size()).then([f = std::move(f), buffer = std::move(buffer)] (size_t bytes_written) {
std::cout << "Wrote " << bytes_written << " bytes to " << f.get_path() << std::endl;
// Close the file asynchronously
return f.close();
});
}).handle_exception([](std::exception_ptr ep) {
// Handle any exceptions during file operations
std::cerr << "Error writing to file: " << seastar::current_exception_better_what(ep) << std::endl;
return seastar::make_exception_future<>(ep);
});
}
// Function to read data asynchronously from a file
seastar::future<seastar::sstring> read_from_file(const seastar::sstring& filename) {
return seastar::open_file_dma(filename, seastar::open_flags::ro).then([filename](seastar::file f) {
// Get file size to allocate buffer
return f.size().then([f = std::move(f), filename](uint64_t size) mutable {
auto buffer = seastar::temporary_buffer<char>::aligned(4096, size);
// Read into the buffer asynchronously
return f.dma_read(buffer.get(), 0, size).then([f = std::move(f), buffer = std::move(buffer)](size_t bytes_read) mutable {
std::cout << "Read " << bytes_read << " bytes from " << f.get_path() << std::endl;
// Close the file asynchronously
return f.close().then([buffer = std::move(buffer)]() mutable {
return seastar::sstring(buffer.begin(), buffer.size());
});
});
});
}).handle_exception([](std::exception_ptr ep) {
std::cerr << "Error reading from file: " << seastar::current_exception_better_what(ep) << std::endl;
return seastar::make_exception_future<seastar::sstring>(ep);
});
}
int main(int argc, char** argv) {
seastar::app_template app;
// Define the application's main function
app.run(argc, argv, [] {
return seastar::make_ready_future().then([] {
seastar::sstring test_data = "Hello, Seastar asynchronous I/O!";
seastar::sstring filename = "test_async_io.txt";
// Chain asynchronous operations: write, then read, then print
return write_to_file(filename, test_data).then([filename] {
return read_from_file(filename);
}).then([](seastar::sstring content) {
std::cout << "File content: " << content << std::endl;
}).handle_exception([](std::exception_ptr ep) {
std::cerr << "Application failed: " << seastar::current_exception_better_what(ep) << std::endl;
return seastar::make_exception_future<>(ep);
});
});
});
return 0;
}
To compile and run this Seastar example:
# Assuming Seastar is installed and SEASTAR_HOME is set
g++ -std=c++17 -Wall -Werror -O2 -I${SEASTAR_HOME}/include -L${SEASTAR_HOME}/build/lib -Wl,-rpath=${SEASTAR_HOME}/build/lib -o async_io async_io.cpp -lseastar -lfmt -lstdc++fs -lboost_program_options -lboost_thread -lboost_system -lboost_filesystem -lboost_chrono -lboost_context -lboost_atomic -lhwloc -latomic -lrt -lm -ldl -lucontext -lnuma -lz -lcryptopp -lgnutls -lprotobuf -ljsoncpp -lcap -luring
./async_io --smp 1 # Run with 1 core
Modern multi-socket servers employ Non-Uniform Memory Access (NUMA) architectures. Memory access times vary depending on whether the memory is local to the CPU accessing it or located on another NUMA node. ScyllaDB is explicitly NUMA-aware. It partitions its memory pools such that each Seastar thread allocates memory from the NUMA node it resides on. This significantly reduces cross-NUMA node memory access, which is slower and consumes inter-socket bandwidth.
io_uring)ScyllaDB bypasses the kernel's block layer for disk I/O whenever possible. It uses asynchronous I/O interfaces like Linux AIO (Asynchronous I/O) or, more recently and preferably, io_uring. io_uring is a modern Linux kernel interface that provides a highly efficient, zero-copy asynchronous I/O mechanism.
Each Seastar core maintains its own io_uring submission and completion queues. This allows direct submission of I/O requests from user-space to the kernel, and direct retrieval of completion events, minimizing context switches and system call overheads. ScyllaDB also implements its own I/O scheduler in user-space, allowing fine-grained control over I/O prioritization and fairness, which is crucial for maintaining low latency under mixed workloads.
// Conceptual illustration of io_uring usage within Seastar (simplified)
// In reality, Seastar abstracts this heavily.
#include <liburing.h>
#include <fcntl.h>
#include <unistd.h>
#include <sys/stat.h>
#include <iostream>
#include <vector>
#include <string>
// This is a simplified, standalone example to demonstrate io_uring concepts.
// Seastar integrates io_uring much more deeply into its reactor model.
const int QUEUE_DEPTH = 64;
const int BLOCK_SIZE = 4096;
int main() {
struct io_uring ring;
int ret = io_uring_queue_init(QUEUE_DEPTH, &ring, 0);
if (ret < 0) {
std::cerr << "io_uring_queue_init: " << strerror(-ret) << std::endl;
return 1;
}
int fd = open("test_io_uring.txt", O_RDWR | O_CREAT | O_TRUNC, 0644);
if (fd < 0) {
std::cerr << "open: " << strerror(errno) << std::endl;
io_uring_queue_exit(&ring);
return 1;
}
std::string write_data = "Hello from io_uring!";
std::vector<char> write_buffer(BLOCK_SIZE);
std::copy(write_data.begin(), write_data.end(), write_buffer.begin());
// Prepare a write request
struct io_uring_sqe *sqe = io_uring_get_sqe(&ring);
if (!sqe) {
std::cerr << "io_uring_get_sqe failed" << std::endl;
close(fd);
io_uring_queue_exit(&ring);
return 1;
}
io_uring_prep_write(sqe, fd, write_buffer.data(), write_buffer.size(), 0);
sqe->user_data = 1; // Unique identifier for this request
// Submit the request
io_uring_submit(&ring);
// Wait for completion
struct io_uring_cqe *cqe;
ret = io_uring_wait_cqe(&ring, &cqe);
if (ret < 0) {
std::cerr << "io_uring_wait_cqe: " << strerror(-ret) << std::endl;
close(fd);
io_uring_queue_exit(&ring);
return 1;
}
if (cqe->res < 0) {
std::cerr << "Write failed: " << strerror(-cqe->res) << std::endl;
} else {
std::cout << "Write completed: " << cqe->res << " bytes" << std::endl;
}
io_uring_cqe_seen(&ring, cqe); // Mark completion as seen
// Prepare a read request
std::vector<char> read_buffer(BLOCK_SIZE);
sqe = io_uring_get_sqe(&ring);
if (!sqe) {
std::cerr << "io_uring_get_sqe failed for read" << std::endl;
close(fd);
io_uring_queue_exit(&ring);
return 1;
}
io_uring_prep_read(sqe, fd, read_buffer.data(), read_buffer.size(), 0);
sqe->user_data = 2; // Unique identifier for this request
// Submit the request
io_uring_submit(&ring);
// Wait for completion
ret = io_uring_wait_cqe(&ring, &cqe);
if (ret < 0) {
std::cerr << "io_uring_wait_cqe for read: " << strerror(-ret) << std::endl;
close(fd);
io_uring_queue_exit(&ring);
return 1;
}
if (cqe->res < 0) {
std::cerr << "Read failed: " << strerror(-cqe->res) << std::endl;
} else {
std::cout << "Read completed: " << cqe->res << " bytes. Content: " << std::string(read_buffer.data(), cqe->res) << std::endl;
}
io_uring_cqe_seen(&ring, cqe);
close(fd);
io_uring_queue_exit(&ring);
return 0;
}
To compile and run this io_uring example:
g++ -std=c++17 -Wall -Werror -O2 -o io_uring_example io_uring_example.cpp -luring
./io_uring_example
Note: io_uring requires a recent Linux kernel (5.1 or newer for basic functionality, 5.8+ for advanced features).
| Feature | ScyllaDB (Seastar) | Apache Cassandra (JVM) |
|---|---|---|
| Execution Model | Thread-per-core, shared-nothing | Multi-threaded, shared-memory |
| Concurrency | Cooperative multitasking (futures) | Preemptive multitasking (OS threads) |
| Language | C++ | Java |
| Memory Management | Manual (NUMA-aware allocators) | JVM GC (Stop-the-world/concurrent) |
| I/O Model | User-space AIO/io_uring, direct | Kernel-buffered AIO/NIO, OS scheduler |
| Network Stack | Optional kernel bypass (DPDK/XDP) | Standard kernel TCP/IP stack |
| Latency Predictability | High (sub-millisecond P99) | Lower (GC pauses, context switches) |
| Resource Utilization | Near 100% CPU, high I/O efficiency | Variable, GC overhead, context switching |
| Startup Time | Fast | Slower (JVM JIT, class loading) |
| Footprint | Smaller | Larger (JVM runtime) |
ScyllaDB's architectural choices directly contribute to its ability to sustain predictable sub-millisecond P99 latencies under high throughput (e.g., 500,000 writes/sec).
io_uring provides direct, low-latency access to storage devices, bypassing kernel overheads and allowing ScyllaDB to prioritize critical I/O operations.Misconfigured CPU Pinning/Isolation:
top showing ScyllaDB processes jumping between cores.isolcpus or cpuset kernel parameters are correctly configured in /etc/default/grub (or equivalent) to isolate cores for ScyllaDB. Use tuned-adm profile scylla on RHEL/CentOS or manually configure irqbalance to avoid routing interrupts to ScyllaDB cores. Verify with lscpu -e and taskset -cp <pid>.NUMA Misalignment:
numa_hit and numa_miss metrics in numastat -m, or perf showing significant remote memory accesses.numactl is installed. Check ScyllaDB logs for NUMA-related warnings. Verify scylla.yaml smp and memory settings are appropriate for your hardware. For example, if you have two NUMA nodes, smp should be half your total cores, and memory should be half your total RAM.Insufficient io_uring / AIO Resources:
iostat showing high await times.io_uring or AIO limits are too low, or the underlying storage is saturated.io_uring, ensure kernel is recent enough. For older AIO, increase fs.aio-max-nr in /etc/sysctl.conf (e.g., fs.aio-max-nr = 1048576). Monitor disk I/O metrics closely. Consider faster storage (NVMe) or more I/O paths.Network Stack Contention (without DPDK/XDP):
netstat -s showing errors, especially on high-throughput nodes.net.core.rmem_max, net.core.wmem_max, net.ipv4.tcp_rmem, net.ipv4.tcp_wmem).Over-provisioning/Under-provisioning Cores:
scylla.yaml smp setting does not match the available isolated cores, or the workload is unevenly distributed.smp to the number of isolated cores dedicated to ScyllaDB. Leave at least one core for the OS and background tasks. Ensure your data model and partition keys lead to an even distribution of data and requests across nodes and cores.A high-traffic e-commerce platform is experiencing unpredictable latency spikes and occasional timeouts with its traditional multi-threaded database, especially during peak sales events. The engineering team is considering migrating to a ScyllaDB-like architecture. Which of the following is the MOST likely root cause of the current system's P99 latency issues that ScyllaDB's architecture directly addresses?
Why does ScyllaDB use C++ instead of a garbage-collected language like Java or Go? ScyllaDB uses C++ to achieve deterministic performance and fine-grained control over system resources. This avoids the unpredictable latency spikes caused by garbage collection pauses (common in Java/Go) and allows for direct memory management, NUMA awareness, and user-space I/O, which are critical for sub-millisecond P99 latency at scale.
How does ScyllaDB handle concurrency without traditional locks or OS threads? ScyllaDB employs a "shared-nothing, thread-per-core" model. Each CPU core runs a dedicated Seastar thread with its own local memory, network, and I/O queues. Concurrency within a core is managed via cooperative multitasking using futures and continuations. Inter-core communication occurs via explicit message passing, avoiding shared memory contention and locks.
What is io_uring and why is it important for ScyllaDB's performance?
io_uring is a modern Linux kernel interface for asynchronous I/O. It allows ScyllaDB to submit and complete I/O requests directly from user-space without system calls or context switches for each operation. This significantly reduces I/O overhead, improves throughput, and lowers latency compared to traditional kernel-buffered I/O or older AIO interfaces.
How does ScyllaDB ensure data locality and NUMA awareness? ScyllaDB's Seastar engine is designed to be NUMA-aware. It partitions its memory pools and I/O queues such that each Seastar thread (pinned to a CPU core) allocates and accesses memory primarily from its local NUMA node. This minimizes slower cross-NUMA node memory access, which is crucial for consistent high performance on multi-socket servers.
Can I run other applications on the same server as ScyllaDB? While technically possible, it is strongly discouraged for production ScyllaDB deployments. ScyllaDB is designed to consume nearly all available CPU, memory, and I/O resources on its dedicated cores for optimal performance. Running other applications on the same server, especially on the same isolated cores, will lead to resource contention, unpredictable latency, and degraded ScyllaDB performance. Dedicated hardware or isolated virtual machines with CPU pinning are recommended.
Clean AI CLI logs, build cron expressions, decode JWTs, and calculate chmod permissions offline.

ScyllaDB vs Apache Cassandra in 2026: P99 latency under write spikes, Java JVM garbage collection pauses vs C++ Seastar, and node count savings.
Read more
Redpanda vs Apache Kafka benchmarked on NVMe drives: thread-per-core Seastar architecture, zero-page-cache latency, and hardware TCO comparison.
Read more
WarpStream vs Redpanda vs Kafka: zero-disk S3 event streaming economics, inter-AZ transfer cost savings, and produce-to-consume latency numbers.
Read more