We upgraded cluster over the weekend, and one of our data nodes crashes about once an hour with the following error :
Node 3: Forced node shutdown completed. Caused by error 2341: 'Internal program error (failed ndbrequire)(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'.
The full error from the error log is :
Time: Monday 16 April 2012 - 09:10:55
Status: Temporary error, restart node
Message: Internal program error (failed ndbrequire) (Internal error, programming error or missing error message, please report a bug)
Error: 2341
Error data: DblqhMain.cpp
Error object: DBLQH (Line: 9785) 0x00000002
Program: ndbd
Pid: 13135
Version: mysql-5.5.20 ndb-7.2.5
Trace: /var/lib/mysql/mysql-cluster/ndb_data/ndb_3_trace.log.3 [t1..t1]
***EOM***
We have rebuilt the node with --initial, but the crashing continues. The cluster was rock solid under 7.2.4, but that version contained a bug #64350 we needed to resolve. Since the error message does not point to a specific problem, I'm not sure where to go with this. Below is the config.ini.
config.ini
[ndb_mgmd]
HostName=db-admin
DataDir=/var/lib/mysql/mysql-cluster/ndb_data
NodeId=1
[ndbd default]
noofreplicas=2
DataDir=/var/lib/mysql/mysql-cluster/ndb_data
MaxNoOfOrderedIndexes=4096
MaxNoOfAttributes=10000
DataMemory=4096M
IndexMemory=500M
MaxNoOfConcurrentOperations=500000
MaxNoOfLocalOperations=550000
MaxNoOfExecutionThreads=8
DiskCheckPointSpeed=20M
[ndbd]
HostName=db1
NodeId=3
[ndbd]
HostName=db2
NodeId=4
[mysqld]
NodeId=51
[mysqld]
NodeId=52
[mysqld]
NodeId=53
Node 3: Forced node shutdown completed. Caused by error 2341: 'Internal program error (failed ndbrequire)(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'.
The full error from the error log is :
Time: Monday 16 April 2012 - 09:10:55
Status: Temporary error, restart node
Message: Internal program error (failed ndbrequire) (Internal error, programming error or missing error message, please report a bug)
Error: 2341
Error data: DblqhMain.cpp
Error object: DBLQH (Line: 9785) 0x00000002
Program: ndbd
Pid: 13135
Version: mysql-5.5.20 ndb-7.2.5
Trace: /var/lib/mysql/mysql-cluster/ndb_data/ndb_3_trace.log.3 [t1..t1]
***EOM***
We have rebuilt the node with --initial, but the crashing continues. The cluster was rock solid under 7.2.4, but that version contained a bug #64350 we needed to resolve. Since the error message does not point to a specific problem, I'm not sure where to go with this. Below is the config.ini.
config.ini
[ndb_mgmd]
HostName=db-admin
DataDir=/var/lib/mysql/mysql-cluster/ndb_data
NodeId=1
[ndbd default]
noofreplicas=2
DataDir=/var/lib/mysql/mysql-cluster/ndb_data
MaxNoOfOrderedIndexes=4096
MaxNoOfAttributes=10000
DataMemory=4096M
IndexMemory=500M
MaxNoOfConcurrentOperations=500000
MaxNoOfLocalOperations=550000
MaxNoOfExecutionThreads=8
DiskCheckPointSpeed=20M
[ndbd]
HostName=db1
NodeId=3
[ndbd]
HostName=db2
NodeId=4
[mysqld]
NodeId=51
[mysqld]
NodeId=52
[mysqld]
NodeId=53