Continual Failure "Forced node shutdown complete" (no replies)

Dear Cluster,

My cluster (7.2.5 on Linux) is starting:

2012-09-19 10:10:17 [ndbd] INFO -- Node started

(Full start log at end of message)

However when I try to access my data I am getting:


2012-09-19 10:10:18 [ndbd] INFO -- /pb2/build/sb_0-5227860-1331723713.63/mysql-cluster-gpl-7.2.5/storage/ndb/src/kernel/blocks/dbacc/DbaccMain.cpp
2012-09-19 10:10:18 [ndbd] INFO -- DBACC (Line: 1052) 0x00000002
2012-09-19 10:10:18 [ndbd] INFO -- Error handler shutting down system
2012-09-19 10:10:18 [ndbd] INFO -- Error handler shutdown completed - exiting
2012-09-19 10:10:18 [ndbd] ALERT -- Node 4: Forced node shutdown completed. Caused by error 2341: 'Internal program error (failed ndbrequire)(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'.


Therefore, (yet again) my cluster is dead.


Can somebody from cluster please tell me what is wrong, and how to keep this thing working for more than a few days at a time?

Regards, Ben Clewett.


2012-09-19 10:16:19 [ndbd] INFO -- Angel pid: 15032 started child: 15033
2012-09-19 10:16:19 [ndbd] INFO -- Configuration fetched from 'localhost:1186', generation: 1
NDBMT: non-mt
2012-09-19 10:16:19 [ndbd] INFO -- NDB Cluster -- DB node 4
2012-09-19 10:16:19 [ndbd] INFO -- mysql-5.5.20 ndb-7.2.5 --
2012-09-19 10:16:19 [ndbd] INFO -- WatchDog timer is set to 6000 ms
2012-09-19 10:16:19 [ndbd] INFO -- numa_set_interleave_mask(numa_all_nodes) : OK
2012-09-19 10:16:19 [ndbd] INFO -- Ndbd_mem_manager::init(1) min: 716Mb initial: 844Mb
Adding 68Mb to ZONE_LO (1,2175)
Instantiating DBSPJ instanceNo=0
Locked to CPU ok
Locked to CPU ok
Locked to CPU ok
Locked to CPU ok
2012-09-19 10:16:21 [ndbd] INFO -- Start initiated (mysql-5.5.20 ndb-7.2.5)
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
Locked to CPU ok
Adding 777Mb to ZONE_LO (2176,24840)
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
Locked to CPU ok
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
Locked to CPU ok
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
Locked to CPU ok
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
Locked to CPU ok
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
Locked to CPU ok
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
Locked to CPU ok
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
Locked to CPU ok
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
Locked to CPU ok
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
Locked to CPU ok
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
Locked to CPU ok
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
Locked to CPU ok
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
Locked to CPU ok
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
Locked to CPU ok
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
Locked to CPU ok
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
Locked to CPU ok
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
Locked to CPU ok
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
Locked to CPU ok
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
Locked to CPU ok
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
Locked to CPU ok
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
Locked to CPU ok
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
Locked to CPU ok
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
Locked to CPU ok
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
Locked to CPU ok
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
Locked to CPU ok
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
Locked to CPU ok
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
Locked to CPU ok
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
Locked to CPU ok
Locked to CPU ok
Locked to CPU ok
WOPool::init(61, 9)
RWPool::init(22, 14)
blockSz: 800, wpp: 8188 -> 624 (76)
RWPool::init(29, 624)
RWPool::init(42, 16)
RWPool::init(62, 12)
RWPool::init(c2, 18)
RWPool::init(e2, 16)
WOPool::init(41, 8)
RWPool::init(82, 12)
RWPool::init(a2, 54)
WOPool::init(21, 10)
blockSz: 1024, wpp: 8188 -> 816 (28)
RWPool::init(68, 816)
2012-09-19 10:16:22 [ndbd] INFO -- Start phase 0 completed
2012-09-19 10:16:22 [ndbd] INFO -- Using locked memory