CouchBase and NUMA

This analysis was performed on CentOS 6.5 and Couchbase 2.5.

NUMA (None Uniform Memory Access) is something which many system administrators often overlooked when configuring memory intensive applications on multi-socket and multi-core machines.
NUMA is a computer memory design where memory access is time dependent because of the physical layout of computer circuitry. I believe Fig 1 provides a good explanation of how remote memory access slow processing time in application which cache large amounts of data in memory. For example, Couchbase aggressively caches all data documents in memory. Each NUMA Node is composed of memory banks and CPU socket(s).

Fig. 1
NUMA

What couchbases’s stance on configuring numactl for couch?
At present Couchbase does not implement any NUMA related optimization, which can be observed by issuing the numactl command. In Fig. 2, NUMA Node 0 has 21,422MB free while Node 1 has 27,385MB free. This results in a memory imbalance across nodes.

To check if your system is affect install numactl.

[root@cb02] yum install numactl

After installing numactl, issue the command “numactl –hardware”
Fig. 2

[root@cb02] numactl --hardware
available: 2 nodes (0-1)
node 0 cpus: 0 2 4 6 8 10 12 14 16 18 20 22
node 0 size: 32722 MB
node 0 free: 21422 MB
node 1 cpus: 1 3 5 7 9 11 13 15 17 19 21 23
node 1 size: 32768 MB
node 1 free: 27385 MB
node distances:
node 0 1
0: 10 20
1: 20 10

On Couchbase node two in the Couchbase cluster the NUMA imbalance is even worse.

[root@cb01] numactl --hardware
available: 2 nodes (0-1)
node 0 cpus: 0 2 4 6 8 10 12 14 16 18 20 22
node 0 size: 32722 MB
node 0 free: 19500 MB
node 1 cpus: 1 3 5 7 9 11 13 15 17 19 21 23
node 1 size: 32768 MB
node 1 free: 27532 MB
node distances:
node 0 1
0: 10 20
1: 20 10

To correct the issue we must modify the couchbase-server startup script to enable NUMA interleave.

vi /etc/init.d/couchbase-server

Line before editing init script couchbase-server

daemon --user couchbase "$DAEMON -- -noinput -detached > /opt/couchbase/var/lib/couchbase/logs/start.log 2>&1"

After adding “numactl –interleave all” to the beginning of the line

daemon --user couchbase "numactl --interleave all $DAEMON -- -noinput -detached > /opt/couchbase/var/lib/couchbase/logs/start.log 2>&1"

From the numactl MAN pages interleaving memory will be allocated in a round robin architecture.
“Set a memory interleave policy. Memory will be allocated using round robin on
nodes. When memory cannot be allocated on the current interleave target fall
back to other nodes.”

After updating the couchbase-server init script issue a restart

[root@cb01] service couchbase-server restart

Rerun numactl to verify that the NUMA memory nodes are in balanace.

[root@cb01] numactl --hardware

Resources: http://support.couchbase.com/entries/44029184-NUMA-Configuration

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s