Hi all
I was doing some testing/benchmarking of a MySQL NDB Cluster through memcached interface, using brutis (http://code.google.com/p/brutis/) to generate memcached transaction and I've seen that when pushing writes to the node upper limit (it reaches aprox 18-20k sets/sec), after a few minutes, it starts throwing
NDB Temporary Error 410: REDO log files overloaded (decrease TimeBetweenLocalCheckpoints or increase NoOfFragmentLogFiles)
I guess this is due to the fact that it cannot commit the REDO log (on disk) to the checkpoint (on disk), due to the massive amount of writes. And if I get this right, this is in spite of the REDO log size; I mean, the REDO log size just moves forward the error point, or it can help you manage a write burst, but not such a high sustained write traffic.
So, I slowed down the writes to about 2k/sec (I'm running on a RAID-1 SSD system capable of 350MB/sec according to hdparm) hoping to reach the optimal write/read point for this hardware...now the test doesn't fail after a couple of minutes but again after like 15 minutes fails with this new, strange error
Memcached Error 29024: Out of Ndb instances, growing freelist
I googled it and I cannot find anything about it, a part from the bzr commit of the error message in Mysql Cluster code, so I'm asking: what's this error?
My config.ini:
[ndbd default]
# Options affecting ndbd processes on all data nodes:
NoOfReplicas=3 # Number of replicas
DataMemory=4000M # How much memory to allocate for data storage
IndexMemory=2000M
MaxNoOfTables=256
MaxNoOfOrderedIndexes=256
MaxNoOfUniqueHashIndexes=128
MaxNoOfConcurrentOperations=200000
MaxNoOfExecutionThreads=13
TimeBetweenLocalCheckpoints=6
NoOfFragmentLogFiles=8
[ndb_mgmd]
# Management process options:
hostname=test-01.backend # Hostname or IP address of MGM node
datadir=/var/lib/mysql-cluster # Directory for MGM node log files
NodeId=1
[ndbd]
hostname=test-01.backend # Hostname or IP address
datadir=/var/lib/mysql/data # Directory for this data node's data files
NodeId=2
[ndbd]
hostname=test-02.backend # Hostname or IP address
datadir=/var/lib/mysql/data # Directory for this data node's data files
NodeId=3
[ndbd]
hostname=test-03.backend # Hostname or IP address
datadir=/var/lib/mysql/data # Directory for this data node's data files
NodeId=4
[mysqld]
NodeId=50
[mysqld]
NodeId=51
[mysqld]
NodeId=52
[mysqld]
NodeId=53
[mysqld]
NodeId=54
[mysqld]
NodeId=55
[mysqld]
NodeId=56
[mysqld]
NodeId=57
[mysqld]
NodeId=58
[mysqld]
NodeId=59
[mysqld]
NodeId=60
[mysqld]
NodeId=61
[mysqld]
NodeId=62
[mysqld]
NodeId=63
[mysqld]
NodeId=64
The emcached command line
# memcached -E /var/lib/mysql/lib/ndb_engine.so -e "connectstring=test-kyoto-01.backend:1186;role=db-only" -u mysql -t 50
The Brutis command line:
# ./brutis -x test-02.backend:11211 -v -P -f 50 -r 20:100 -s 600 -p "test4-" -n 10000000 -z 40000 -a S:R
Thanks!
I was doing some testing/benchmarking of a MySQL NDB Cluster through memcached interface, using brutis (http://code.google.com/p/brutis/) to generate memcached transaction and I've seen that when pushing writes to the node upper limit (it reaches aprox 18-20k sets/sec), after a few minutes, it starts throwing
NDB Temporary Error 410: REDO log files overloaded (decrease TimeBetweenLocalCheckpoints or increase NoOfFragmentLogFiles)
I guess this is due to the fact that it cannot commit the REDO log (on disk) to the checkpoint (on disk), due to the massive amount of writes. And if I get this right, this is in spite of the REDO log size; I mean, the REDO log size just moves forward the error point, or it can help you manage a write burst, but not such a high sustained write traffic.
So, I slowed down the writes to about 2k/sec (I'm running on a RAID-1 SSD system capable of 350MB/sec according to hdparm) hoping to reach the optimal write/read point for this hardware...now the test doesn't fail after a couple of minutes but again after like 15 minutes fails with this new, strange error
Memcached Error 29024: Out of Ndb instances, growing freelist
I googled it and I cannot find anything about it, a part from the bzr commit of the error message in Mysql Cluster code, so I'm asking: what's this error?
My config.ini:
[ndbd default]
# Options affecting ndbd processes on all data nodes:
NoOfReplicas=3 # Number of replicas
DataMemory=4000M # How much memory to allocate for data storage
IndexMemory=2000M
MaxNoOfTables=256
MaxNoOfOrderedIndexes=256
MaxNoOfUniqueHashIndexes=128
MaxNoOfConcurrentOperations=200000
MaxNoOfExecutionThreads=13
TimeBetweenLocalCheckpoints=6
NoOfFragmentLogFiles=8
[ndb_mgmd]
# Management process options:
hostname=test-01.backend # Hostname or IP address of MGM node
datadir=/var/lib/mysql-cluster # Directory for MGM node log files
NodeId=1
[ndbd]
hostname=test-01.backend # Hostname or IP address
datadir=/var/lib/mysql/data # Directory for this data node's data files
NodeId=2
[ndbd]
hostname=test-02.backend # Hostname or IP address
datadir=/var/lib/mysql/data # Directory for this data node's data files
NodeId=3
[ndbd]
hostname=test-03.backend # Hostname or IP address
datadir=/var/lib/mysql/data # Directory for this data node's data files
NodeId=4
[mysqld]
NodeId=50
[mysqld]
NodeId=51
[mysqld]
NodeId=52
[mysqld]
NodeId=53
[mysqld]
NodeId=54
[mysqld]
NodeId=55
[mysqld]
NodeId=56
[mysqld]
NodeId=57
[mysqld]
NodeId=58
[mysqld]
NodeId=59
[mysqld]
NodeId=60
[mysqld]
NodeId=61
[mysqld]
NodeId=62
[mysqld]
NodeId=63
[mysqld]
NodeId=64
The emcached command line
# memcached -E /var/lib/mysql/lib/ndb_engine.so -e "connectstring=test-kyoto-01.backend:1186;role=db-only" -u mysql -t 50
The Brutis command line:
# ./brutis -x test-02.backend:11211 -v -P -f 50 -r 20:100 -s 600 -p "test4-" -n 10000000 -z 40000 -a S:R
Thanks!