Quantcast
Channel: MySQL Forums - NDB clusters
Viewing all articles
Browse latest Browse all 1562

Data node stuck in phase 4 with error code (no replies)

$
0
0
Hi, I think I need some help to restart my data nodes in cluster. I have a cluster with 4 data nodes and hundreds of millions record inserted. But somehow I couldn't successfully start the cluster now due to failure in data nodes, as show below. Can anyone provide any clue why this happened?

2014-10-06 19:10:33 [ndbd] INFO -- Start phase 0 completed
2014-10-06 19:10:36 [ndbd] INFO -- findNeighbours from: 2202 old (left: 65535 right: 65535) new (12 11)
2014-10-06 19:10:36 [ndbd] INFO -- Start phase 1 completed
2014-10-06 19:10:36 [ndbd] INFO -- findNeighbours from: 2114 old (left: 12 right: 11) new (12 14)
2014-10-06 19:10:36 [ndbd] INFO -- Start phase 2 completed
2014-10-06 19:10:36 [ndbd] INFO -- Start phase 3 completed
restartCreateObj(1) file: 1
restartCreateObj(2) file: 1
restartCreateObj(3) file: 1
restartCreateObj(4) file: 1
restartCreateObj(5) file: 1
restartCreateObj(6) file: 1
restartCreateObj(7) file: 1
restartCreateObj(9) file: 1
restartCreateObj(11) file: 1
restartCreateObj(12) file: 1
restartCreateObj(13) file: 1
restartCreateObj(15) file: 1
restartCreateObj(17) file: 1
restartCreateObj(19) file: 1
restartCreateObj(21) file: 1
restartCreateObj(23) file: 1
restartCreateObj(25) file: 1
restartCreateObj(27) file: 1
restartCreateObj(29) file: 1
restartCreateObj(31) file: 1
restartCreateObj(33) file: 1
restartCreateObj(39) file: 1
restartCreateObj(41) file: 1
restartCreateObj(43) file: 1
restartCreateObj(50) file: 1
restartCreateObj(51) file: 1
restartCreateObj(8) file: 1
restartCreateObj(10) file: 1
restartCreateObj(14) file: 1
restartCreateObj(16) file: 1
restartCreateObj(18) file: 1
restartCreateObj(20) file: 1
restartCreateObj(22) file: 1
restartCreateObj(24) file: 1
restartCreateObj(26) file: 1
restartCreateObj(28) file: 1
restartCreateObj(30) file: 1
restartCreateObj(32) file: 1
restartCreateObj(34) file: 1
restartCreateObj(35) file: 1
restartCreateObj(36) file: 1
restartCreateObj(37) file: 1
restartCreateObj(38) file: 1
restartCreateObj(40) file: 1
restartCreateObj(42) file: 1
restartCreateObj(44) file: 1
restartCreateObj(45) file: 1
restartCreateObj(46) file: 1
restartCreateObj(47) file: 1
restartCreateObj(48) file: 1
restartCreateObj(49) file: 1
restartCreateObj(52) file: 1
Using 1 fragments per node
execSTART_RECREQ chaning srnodes from 0000000000007800 to 0000000000002800
RESTORE table: 2 540 rows applied
RESTORE table: 2 481 rows applied
RESTORE table: 3 6 rows applied
RESTORE table: 3 4 rows applied
RESTORE table: 4 8 rows applied
RESTORE table: 4 4 rows applied
RESTORE table: 5 8 rows applied
RESTORE table: 5 4 rows applied
RESTORE table: 6 0 rows applied
RESTORE table: 6 0 rows applied
RESTORE table: 7 70465 rows applied
RESTORE table: 7 69722 rows applied
RESTORE table: 9 62948 rows applied
RESTORE table: 9 62300 rows applied
RESTORE table: 11 0 rows applied
RESTORE table: 11 0 rows applied
RESTORE table: 12 0 rows applied
RESTORE table: 12 0 rows applied
RESTORE table: 13 71542 rows applied
RESTORE table: 13 70846 rows applied
RESTORE table: 15 0 rows applied
RESTORE table: 15 0 rows applied
RESTORE table: 17 71624 rows applied
RESTORE table: 17 70907 rows applied
RESTORE table: 19 71624 rows applied
RESTORE table: 19 70907 rows applied
RESTORE table: 21 71542 rows applied
RESTORE table: 21 70846 rows applied
RESTORE table: 23 67152 rows applied
RESTORE table: 23 66504 rows applied
RESTORE table: 25 55916 rows applied
RESTORE table: 25 55465 rows applied
RESTORE table: 27 65815 rows applied
RESTORE table: 27 65136 rows applied
RESTORE table: 29 863 rows applied
RESTORE table: 29 832 rows applied
RESTORE table: 31 0 rows applied
RESTORE table: 31 0 rows applied
2014-10-06 19:11:35 [ndbd] INFO -- RESTORE: File system read failed. OS errno: 87
2014-10-06 19:11:35 [ndbd] INFO -- RESTORE (Line: 3656) 0x00000002
2014-10-06 19:11:35 [ndbd] INFO -- Error handler shutting down system
2014-10-06 19:11:35 [ndbd] INFO -- Error handler shutdown completed - exiting
2014-10-06 19:11:41 [ndbd] ALERT -- Node 13: Forced node shutdown completed. Occured during startphase 4. Caused by error 2813: 'Unknown file system error(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'.

Viewing all articles
Browse latest Browse all 1562

Trending Articles