Data node deletion in NDB (1 reply)

November 30, 2021, 2:36 pm

≫ Next: Procedure for enabling auditing in MySQL 7.6.17(community version) (1 reply)

$

0

0

Hi Team,

I recently added 2 data nodes in ndb cluster through 'ndb_mgm' utility, Now I'm looking to delete that newly added node from the cluster.

Someone Kindly helps.

Cluster version 8.0.26.

Regards,

---

Procedure for enabling auditing in MySQL 7.6.17(community version) (1 reply)

January 5, 2022, 7:03 am

≫ Next: NDB data node failed and sent another data node into a crash loop (2 replies)

≪ Previous: Data node deletion in NDB (1 reply)

$

0

0

We use MySQL 7.6.17(community version) and would like to know the procedure for enabling the auditing feature in this version

NDB data node failed and sent another data node into a crash loop (2 replies)

January 27, 2022, 6:11 am

≫ Next: Got error 738 'Record too big' from NDB (1 reply)

≪ Previous: Procedure for enabling auditing in MySQL 7.6.17(community version) (1 reply)

$

0

0

I have an NDB cluster running 3 data nodes and 3 management nodes.

It appears that an error occured on 1 data node which caused it to restart

Node 19:
2022-01-15 04:20:46 [ndbd] INFO -- findNeighbours from: 2905 old (left: 17 right: 17) new (17 18)
2022-01-15 04:20:47 [ndbd] INFO -- NR Status: node=18,OLD=Node failure handling complete,NEW=All nodes permitted us
2022-01-15 04:20:47 [ndbd] INFO -- Switch to 17 multi trp for node 18
2022-01-15 04:21:24 [ndbd] INFO -- NR Status: node=18,OLD=All nodes permitted us,NEW=Include node in LCP/GCP protocols
2022-01-15 04:21:24 [ndbd] INFO -- NR Status: node=18,OLD=Include node in LCP/GCP protocols,NEW=Synchronize start node with live nodes
For help with below stacktrace consult:
https://dev.mysql.com/doc/refman/en/using-stack-trace.html
Also note that stack_bottom and thread_stack will always show up as zero.
stack_bottom = 0 thread_stack 0x0
/usr/sbin/ndbmtd(my_print_stacktrace(unsigned char const*, unsigned long)+0x3d) [0x8c676d]
/usr/sbin/ndbmtd(ErrorReporter::handleError(int, char const*, char const*, NdbShutdownType)+0x2f) [0x81f59f]
/usr/sbin/ndbmtd(SimulatedBlock::progError(int, int, char const*, char const*) const+0xf9) [0x887ca9]
/usr/sbin/ndbmtd(Dblqh::execCOPY_FRAGREQ(Signal*)+0xa29) [0x65f949]
/usr/sbin/ndbmtd() [0x89604c]
/usr/sbin/ndbmtd() [0x89bc98]
/usr/sbin/ndbmtd(mt_job_thread_main+0x230) [0x8a11e0]
/usr/sbin/ndbmtd() [0x869366]
/lib64/libpthread.so.0(+0x7ea5) [0x7f7b742a3ea5]
/lib64/libc.so.6(clone+0x6d) [0x7f7b72ccb96d]
2022-01-15 04:21:25 [ndbd] INFO -- /var/lib/pb2/sb_1-2918142-1619218179.52/rpm/BUILD/mysql-cluster-com-8.0.25/mysql-cluster-com-8.0.25/storage/ndb/src/kernel/blocks/dblqh/DblqhMain.cpp
2022-01-15 04:21:25 [ndbd] INFO -- DBLQH (Line: 19166) 0x00000002 Check getFragmentrec(fragId) failed
2022-01-15 04:21:25 [ndbd] INFO -- Error handler shutting down system
2022-01-15 04:21:26 [ndbd] ALERT -- Node 19: Forced node shutdown completed. Caused by error 2341: 'Internal program error (failed ndbrequire)(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'.
2022-01-15 04:21:27 [ndbd] INFO -- Angel pid: 34452 started child: 34453


Another data node then failed at the same time and was put into a crash loop

Node 18:
2022-01-15 04:21:25 [ndbd] INFO -- LDM(8): Completed copy of fragment T175F3. Changed +0/-0 rows, 0 bytes. 0 pct churn to 0 rows.
2022-01-15 04:21:26 [ndbd] INFO -- Node 19 disconnected in state: 0
For help with below stacktrace consult:
https://dev.mysql.com/doc/refman/en/using-stack-trace.html
Also note that stack_bottom and thread_stack will always show up as zero.
stack_bottom = 0 thread_stack 0x0
/usr/sbin/ndbmtd(my_print_stacktrace(unsigned char const*, unsigned long)+0x3d) [0x8c676d]
/usr/sbin/ndbmtd(ErrorReporter::handleError(int, char const*, char const*, NdbShutdownType)+0x2f) [0x81f59f]
/usr/sbin/ndbmtd(SimulatedBlock::progError(int, int, char const*, char const*) const+0xf9) [0x887ca9]
/usr/sbin/ndbmtd(Qmgr::execDISCONNECT_REP(Signal*)+0x21f) [0x79be7f]
/usr/sbin/ndbmtd() [0x89604c]
/usr/sbin/ndbmtd() [0x89bc05]
/usr/sbin/ndbmtd(mt_job_thread_main+0x4c9) [0x8a1479]
/usr/sbin/ndbmtd() [0x869366]
/lib64/libpthread.so.0(+0x7ea5) [0x7efd8a3e9ea5]
/lib64/libc.so.6(clone+0x6d) [0x7efd88e1196d]
2022-01-15 04:21:26 [ndbd] INFO -- Node 19 disconnected in state: 0
2022-01-15 04:21:26 [ndbd] INFO -- Node 19 disconnected in phase: 3
2022-01-15 04:21:26 [ndbd] INFO -- QMGR (Line: 4245) 0x00000002
2022-01-15 04:21:26 [ndbd] INFO -- Error handler shutting down system
2022-01-15 04:21:26 [ndbd] ALERT -- Node 18: Forced node shutdown completed. Occurred during startphase 5. Caused by error 2308: 'Another node failed during system restart, please investigate error(s) on other node(s)(Restart error). Temporary error, restart node'.
2022-01-15 04:21:27 [ndbd] INFO -- Angel pid: 14167 started child: 14168

It appears that the 2 nodes were caught in some sort of contention where one would crash and then the other would crash

Node 19:
2022-01-15 13:33:22 [ndbd] INFO -- Node 18 disconnected in state: 0
For help with below stacktrace consult:
https://dev.mysql.com/doc/refman/en/using-stack-trace.html
Also note that stack_bottom and thread_stack will always show up as zero.
stack_bottom = 0 thread_stack 0x0
/usr/sbin/ndbmtd(my_print_stacktrace(unsigned char const*, unsigned long)+0x3d) [0x8c676d]
/usr/sbin/ndbmtd(ErrorReporter::handleError(int, char const*, char const*, NdbShutdownType)+0x2f) [0x81f59f]
/usr/sbin/ndbmtd(SimulatedBlock::progError(int, int, char const*, char const*) const+0xf9) [0x887ca9]
/usr/sbin/ndbmtd(Qmgr::failReportLab(Signal*, unsigned short, FailRep::FailCause, unsigned short)+0x96d) [0x7956ad]
/usr/sbin/ndbmtd() [0x89604c]
/usr/sbin/ndbmtd() [0x89bc05]
/usr/sbin/ndbmtd(mt_job_thread_main+0x4c9) [0x8a1479]
/usr/sbin/ndbmtd() [0x869366]
/lib64/libpthread.so.0(+0x7ea5) [0x7ff4b634bea5]
/lib64/libc.so.6(clone+0x6d) [0x7ff4b4d7396d]
2022-01-15 13:33:22 [ndbd] INFO -- Node 18 failed
2022-01-15 13:33:22 [ndbd] INFO -- QMGR (Line: 5039) 0x00000002
2022-01-15 13:33:22 [ndbd] INFO -- Error handler shutting down system
2022-01-15 13:33:22 [ndbd] ALERT -- Node 19: Forced node shutdown completed. Occurred during startphase 2. Caused by error 2308: 'Another node failed during system restart, please investigate error(s) on other node(s)(Restart error). Temporary error, restart node'.

Node 18:
22-01-15 13:33:02 [ndbd] INFO -- (16), tab(6,3), lcpNo: 65535, m_max_restorable_gci: 2429, crestartNewestGci: 2430, srStartGci: 0
For help with below stacktrace consult:
https://dev.mysql.com/doc/refman/en/using-stack-trace.html
Also note that stack_bottom and thread_stack will always show up as zero.
stack_bottom = 0 thread_stack 0x0
stack_bottom = 0 thread_stack 0x0
/usr/sbin/ndbmtd(my_print_stacktrace(unsigned char const*, unsigned long)+0x3d) [0x8c676d]
/usr/sbin/ndbmtd(ErrorReporter::handleError(int, char const*, char const*, NdbShutdownType)+0x2f) [0x81f59f]
/usr/sbin/ndbmtd(SimulatedBlock::progError(int, int, char const*, char const*) const+0xf9) [0x887ca9]
/usr/sbin/ndbmtd(Dblqh::send_restore_lcp(Signal*)+0x9d9) [0x624b59]
/usr/sbin/ndbmtd() [0x89604c]
/usr/sbin/ndbmtd() [0x89bc98]
/usr/sbin/ndbmtd(mt_job_thread_main+0x4c9) [0x8a1479]
/usr/sbin/ndbmtd() [0x869366]
/lib64/libpthread.so.0(+0x7ea5) [0x7fe8c2222ea5]
/lib64/libc.so.6(clone+0x6d) [0x7fe8c0c4a96d]
/usr/sbin/ndbmtd(my_print_stacktrace(unsigned char const*, unsigned long)+0x3d) [0x8c676d]
/usr/sbin/ndbmtd(ErrorReporter::handleError(int, char const*, char const*, NdbShutdownType)+0x2f) [0x81f59f]
/usr/sbin/ndbmtd(SimulatedBlock::progError(int, int, char const*, char const*) const+0xf9) [0x887ca9]
/usr/sbin/ndbmtd(Dblqh::send_restore_lcp(Signal*)+0x9d9) [0x624b59]
/usr/sbin/ndbmtd() [0x89604c]
/usr/sbin/ndbmtd() [0x89bc98]
/usr/sbin/ndbmtd(mt_job_thread_main+0x4c9) [0x8a1479]
/usr/sbin/ndbmtd() [0x869366]
/lib64/libpthread.so.0(+0x7ea5) [0x7fe8c2222ea5]
/lib64/libc.so.6(clone+0x6d) [0x7fe8c0c4a96d]
/usr/sbin/ndbmtd(my_print_stacktrace(unsigned char const*, unsigned long)+0x3d) [0x8c676d]
/usr/sbin/ndbmtd(ErrorReporter::handleError(int, char const*, char const*, NdbShutdownType)+0x2f) [0x81f59f]
/usr/sbin/ndbmtd(SimulatedBlock::progError(int, int, char const*, char const*) const+0xf9) [0x887ca9]
/usr/sbin/ndbmtd(Dblqh::send_restore_lcp(Signal*)+0x9d9) [0x624b59]
/usr/sbin/ndbmtd() [0x89604c]
/usr/sbin/ndbmtd() [0x89bc98]
/usr/sbin/ndbmtd(mt_job_thread_main+0x4c9) [0x8a1479]
/usr/sbin/ndbmtd() [0x869366]
/lib64/libpthread.so.0(+0x7ea5) [0x7fe8c2222ea5]
/lib64/libc.so.6(clone+0x6d) [0x7fe8c0c4a96d]
/usr/sbin/ndbmtd(my_print_stacktrace(unsigned char const*, unsigned long)+0x3d) [0x8c676d]
/usr/sbin/ndbmtd(ErrorReporter::handleError(int, char const*, char const*, NdbShutdownType)+0x2f) [0x81f59f]
/usr/sbin/ndbmtd(SimulatedBlock::progError(int, int, char const*, char const*) const+0xf9) [0x887ca9]
/usr/sbin/ndbmtd(Dblqh::send_restore_lcp(Signal*)+0x9d9) [0x624b59]
/usr/sbin/ndbmtd() [0x89604c]
/usr/sbin/ndbmtd() [0x89bc98]
/usr/sbin/ndbmtd(mt_job_thread_main+0x4c9) [0x8a1479]
/usr/sbin/ndbmtd() [0x869366]
/lib64/libpthread.so.0(+0x7ea5) [0x7fe8c2222ea5]
/lib64/libc.so.6(clone+0x6d) [0x7fe8c0c4a96d]
/usr/sbin/ndbmtd(my_print_stacktrace(unsigned char const*, unsigned long)+0x3d) [0x8c676d]
/usr/sbin/ndbmtd(ErrorReporter::handleError(int, char const*, char const*, NdbShutdownType)+0x2f) [0x81f59f]
/usr/sbin/ndbmtd(SimulatedBlock::progError(int, int, char const*, char const*) const+0xf9) [0x887ca9]
/usr/sbin/ndbmtd(Dblqh::send_restore_lcp(Signal*)+0x9d9) [0x624b59]
/usr/sbin/ndbmtd() [0x89604c]
/usr/sbin/ndbmtd() [0x89bc98]
/usr/sbin/ndbmtd(mt_job_thread_main+0x4c9) [0x8a1479]
/usr/sbin/ndbmtd() [0x869366]
/lib64/libpthread.so.0(+0x7ea5) [0x7fe8c2222ea5]
/lib64/libc.so.6(clone+0x6d) [0x7fe8c0c4a96d]
/usr/sbin/ndbmtd(my_print_stacktrace(unsigned char const*, unsigned long)+0x3d) [0x8c676d]
/usr/sbin/ndbmtd(ErrorReporter::handleError(int, char const*, char const*, NdbShutdownType)+0x2f) [0x81f59f]
/usr/sbin/ndbmtd(SimulatedBlock::progError(int, int, char const*, char const*) const+0xf9) [0x887ca9]
/usr/sbin/ndbmtd(Dblqh::send_restore_lcp(Signal*)+0x9d9) [0x624b59]
/usr/sbin/ndbmtd() [0x89604c]
/usr/sbin/ndbmtd() [0x89bc98]
/usr/sbin/ndbmtd(mt_job_thread_main+0x4c9) [0x8a1479]
/usr/sbin/ndbmtd() [0x869366]
/lib64/libpthread.so.0(+0x7ea5) [0x7fe8c2222ea5]
/lib64/libc.so.6(clone+0x6d) [0x7fe8c0c4a96d]
2022-01-15 13:33:02 [ndbd] INFO -- /var/lib/pb2/sb_1-2918142-1619218179.52/rpm/BUILD/mysql-cluster-com-8.0.25/mysql-cluster-com-8.0.25/storage/ndb/src/kernel/blocks/dblqh/DblqhMain.cpp
2022-01-15 13:33:02 [ndbd] INFO -- DBLQH (Line: 27173) 0x00000002 Check c_local_sysfile.m_max_restorable_gci >= crestartNewestGci failed
2022-01-15 13:33:02 [ndbd] INFO -- Error handler shutting down system
2022-01-15 13:33:02 [ndbd] ALERT -- Node 18: Forced node shutdown completed. Occurred during startphase 5. Caused by error 2341: 'Internal program error (failed ndbrequire)(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'.

Eventually Node 19 was able to correct itself and start properly
Node 18 however was stuck in a crash loop and eventually taken offline

Can you advise on how to recover from this state?

Got error 738 'Record too big' from NDB (1 reply)

February 9, 2022, 7:36 am

≫ Next: Can not restart NDB cluster - Insufficent nodes (no replies)

≪ Previous: NDB data node failed and sent another data node into a crash loop (2 replies)

$

0

0

Hello. I am setting up a web application, and when the installer creates the database, it fails on the step below.

[349][-none-][FATAL] Error creating table: oauth2tokens: Query Failed: CREATE TABLE oauth2tokens (`id` char(36) NOT NULL ,`name` varchar(255) NULL ,`date_entered` datetime NULL ,`date_modified` datetime NULL ,`modified_user_id` char(36) NULL ,`created_by` char(36) NULL ,`description` text NULL ,`deleted` bool DEFAULT '0' NULL ,`token_is_revoked` bool NULL ,`token_type` varchar(255) NULL ,`access_token_expires` datetime NULL ,`access_token` varchar(4000) NULL ,`refresh_token` varchar(4000) NULL ,`refresh_token_expires` datetime NULL ,`grant_type` varchar(255) NULL ,`state` varchar(1024) NULL ,`client` char(36) NULL ,`assigned_user_id` char(36) NULL , PRIMARY KEY (id)) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci: MySQL error 1005: Can't create table 'oauth2tokens' (use SHOW WARNINGS for more info).

After I run SHOW WARNINGS, the error is as below.
Got error 738 'Record too big' from NDB

Please suggest any fixes so I can finish setting up our new web application. Thank you.

Can not restart NDB cluster - Insufficent nodes (no replies)

February 23, 2022, 3:59 am

≫ Next: NDB Cluster Source code (3 replies)

≪ Previous: Got error 738 'Record too big' from NDB (1 reply)

$

0

0

made a rolling restart and this happened while restarting data nodes. no problem with management nodes, so there should not be any issue with config file
```
Status: Temporary error, restart node
Message: Insufficent nodes for system restart (Restart error)
Error: 2353
Error data: Unable to start missing node group! starting: 0000000000000000000000000000000000000400 (missing working fs for: 0000000000000000000000000000000000000000)
Error object: QMGR (Line: 2515) 0x00000002
Program: ndbmtd
Pid: 36197 thr: 0
Version: mysql-8.0.27 ndb-8.0.27
Trace file name: ndb_10_trace.log.10
Trace file path: /var/lib/mysql/data/ndb_10_trace.log.10 [t1..t4]
***EOM***
```
I tried also --nowait-nodes.
also tied to make nodegoups same time while starting nodes but nodes shits down before I am able to, also management gives same error
```
ndb_mgm> Node 11: Forced node shutdown completed. Occurred during startphase 1. Caused by error 2353: 'Insufficent nodes for system restart(Restart error). Temporary error, restart node'.
Node 10: Forced node shutdown completed. Occurred during startphase 1. Caused by error 2308: 'Another node failed during system restart, please investigate error(s) on other node(s)(Restart error). Temporary error, restart node'.
Node 20: Forced node shutdown completed. Occurred during startphase 1. Caused by error 2353: 'Insufficent nodes for system restart(Restart error). Temporary error, restart node'.
Node 21: Forced node shutdown completed. Occurred during startphase 1. Caused by error 2353: 'Insufficent nodes for system restart(Restart error). Temporary error, restart node'.
```
```
ndb_mgm> CREATE NODEGROUP 10,11
* 5005: Error
* 5005-Send to process or receive failed.: Permanent error: Application error
```
I can not start as --initial I have data there that can not be removed

I need urgent help, how to start datanodes without loosing data.
thank you advanced

NDB Cluster Source code (3 replies)

April 18, 2022, 6:56 am

≫ Next: SQLSTATE[HY000]: General error: 1296 Got error 839 'Illegal null attribute' from NDBCLUSTER [updated] (no replies)

≪ Previous: Can not restart NDB cluster - Insufficent nodes (no replies)

$

0

0

Hi I am looking for how the ndb cluster features set were implemented by MySQL. Thus I was looking for the source code. For example: I want to look into synchronization of data between nodes.

I tried looking into following for source code:
For instance, I checked the cluster rpm files.

Can anyone help me in where I can find source code for these?

Thanks
Sandy P.

SQLSTATE[HY000]: General error: 1296 Got error 839 'Illegal null attribute' from NDBCLUSTER [updated] (no replies)

May 26, 2022, 12:33 am

≫ Next: Mysql NDB cluster - Big import fail (no replies)

≪ Previous: NDB Cluster Source code (3 replies)

$

0

0

I was trying to insert a record into a newly created table while got this error message, can anyone tell me why that happened?


Here is my table source:
CREATE TABLE `supcs_recordings` (
`id` bigint unsigned NOT NULL AUTO_INCREMENT,
`project_id` bigint NOT NULL,
`created_at` timestamp NULL DEFAULT NULL,
`updated_at` timestamp NULL DEFAULT NULL,
`updated_by` varchar(64) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`deleted_at` timestamp NULL DEFAULT NULL,
`TYPE` varchar(64) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`owner` varchar(64) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`picked_at` timestamp NULL DEFAULT NULL,
`picked_from` tinyint NOT NULL DEFAULT '0',
`checked` tinyint NOT NULL DEFAULT '0',
`upload_batch` varchar(64) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`uploaded_by` varchar(64) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`uploaded_at` timestamp NULL DEFAULT NULL,
`CATEGORY` varchar(64) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`WEEK` varchar(64) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`DATE_CALL` varchar(64) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`DTIME` varchar(64) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`DURATION` varchar(64) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`CONTRACT` varchar(64) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`CPD` varchar(64) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`REC_SKILL` varchar(64) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`CON_PHASE` varchar(64) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`TASK_TYPE` varchar(64) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`COUNTER_PARTY` varchar(64) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`PH` varchar(64) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`SITE` varchar(64) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`AGENT` varchar(64) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`EMP_ID` varchar(64) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`BUTL` varchar(64) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`TL_SKILL` varchar(64) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`BUSV` varchar(64) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`DD` varchar(64) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`ANS_PARTY` varchar(64) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`CALL_RESULT` varchar(64) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`NOPAY_REASON` varchar(64) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`PTP` varchar(64) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`PH_FROM_NOTE` varchar(64) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`GUID` varchar(64) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`RELEASED_BY_AGENT` varchar(64) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`NEW_NUM` varchar(64) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`AMT` varchar(64) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`INSURED` varchar(64) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`INB_OTB_MAN` varchar(64) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`CONTRACT_NAME` varchar(64) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`SKP_CONTRACT` varchar(64) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`ID_CUID` varchar(64) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`TEXT_IDENTIFICATION` varchar(64) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`CODE_CREDIT_STATUS` varchar(64) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`ID_GAD` varchar(64) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=ndbcluster AUTO_INCREMENT=20 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci

Mysql NDB cluster - Big import fail (no replies)

May 31, 2022, 4:40 am

≫ Next: Distributed MySQL (NDB) database read and write lock (no replies)

≪ Previous: SQLSTATE[HY000]: General error: 1296 Got error 839 'Illegal null attribute' from NDBCLUSTER [updated] (no replies)

$

0

0

Hi

I'm having some issues trying to import a big (12 GB) mysql dump. The dump is from an internal mysql database (no ndb) and I'm trying to insert it to a new mysql ndb cluster (changed all sentences ENGINE=INNODB to ENGINE=NDBCLUSTER).

I've had different issues, some of the errors trying to import were fixed changing configuration parameters, but some others don't. In fact, almost the configuration parameters have been changed as a consequence of an error fired.
For example, the last one that (randomly) appears is:

ERROR 1297 (HY000) at line 9831: Got temporary error 4010 'Node failure caused abort of transaction' from NDBCLUSTER
Warning (Code 1297): Got temporary error 4010 'Node failure caused abort of transaction' from NDB
Error (Code 1297): Got temporary error 4010 'Node failure caused abort of transaction' from NDBCLUSTER
Error (Code 1180): Got error 4010 - 'Unknown error 4010' during COMMIT

Every time an error is fired seems is not related to out of ram, out of disk, or other operating system related issues.

It seems to my that this ndb cluster is too hard to configure, compared to a simple mysql database (no configuration at all and it simply works).

Is there any standard configuration that may fit for this "big" environments? (and I said "big" because my database dump is 12 GB which is not too big).
Or is there any configuration parameters related to these mysql import problems?

This dump can be inserted without issues on a mysql database (no cluster)


My environment is:

- one management node (16 GB RAM)

- two ndb nodes (32 GB RAM each)

- one mysql server node (16 GB RAM)

All mysql cluster versions software are 8.0.29.

config.ini:


[ndbd default]
# Options affecting ndbd processes on all data nodes:
NoOfReplicas=2

#MaxNoOfConcurrentOperations must be greater than MaxNoOfConcurrentTransactions
MaxNoOfConcurrentOperations = 1629496
MaxNoOfConcurrentTransactions = 1629496
MaxDMLOperationsPerTransaction = 1629400
#MaxNoOfConcurrentOperations = 4294967039
#MaxNoOfConcurrentTransactions = 4294900000
#MaxDMLOperationsPerTransaction = 4294967295

DataMemory = 16000M # Memory allocate for data storage
IndexMemory = 2048M # Memory allocate for index storage

#temporal para import inicial
NoOfFragmentLogFiles = 1000

SharedGlobalMemory = 10G

MaxNoOfOrderedIndexes = 1000
MaxNoOfAttributes = 100000

TransactionDeadlockDetectionTimeout = 10000

#para evitar REDO log files overloaded error
TimeBetweenLocalCheckpoints = 2

FragmentLogFileSize = 32M

[tcp default]
#hay un error de Out of SendBufferMemory in sendSignal
SendBufferMemory=16M
ReceiveBufferMemory=16M



[ndb_mgmd]
# Management process options:
hostname=wocu-ha-mysqlcluster-dev-1 # Hostname of the manager
datadir=/var/lib/mysql-cluster # Directory for the log files


[ndbd]
hostname=wocu-ha-mysqlcluster-dev-2 # Hostname/IP of the first data node
NodeId=2 # Node ID for this data node
datadir=/var/lib/mysql-cluster/data # Remote directory for the data files


[ndbd]
hostname=wocu-ha-mysqlcluster-dev-3 # Hostname/IP of the second data node
NodeId=3 # Node ID for this data node
datadir=/var/lib/mysql-cluster/data # Remote directory for the data files


[mysqld]
# SQL node options:
hostname=wocu-ha-mysqlcluster-dev-4



Thanks

---

Distributed MySQL (NDB) database read and write lock (no replies)

June 21, 2022, 5:36 am

≫ Next: MySQL NDB Cluster 8.0.30: Release Notes (no replies)

≪ Previous: Mysql NDB cluster - Big import fail (no replies)

$

0

0

When using the MySQL NDB Cluster distributed database, the following general problem has arisen: To insert a new Record / Row, the last row in the database should be read, see if it is compatible with the row to be inserted and then insert the new row.
However, the problem arises that in the meantime another instance also runs through the same process and then at the end both new rows have checked to the same last record to be compatible and are inserted.
Is there a way to block this with a locking process?

So that one instance gets the last row/ record via JDBC in Java and locks this row for reading, then checks the object and then insert the new record and now a second instance can run through the same process?

Thanks for your answers.

MySQL NDB Cluster 8.0.30: Release Notes (no replies)

July 26, 2022, 4:16 pm

≫ Next: How to build NDB clusters from source code in arm ubuntu (1 reply)

≪ Previous: Distributed MySQL (NDB) database read and write lock (no replies)

$

0

0

MySQL NDB Cluster 8.0.30: Release Notes
- https://dev.mysql.com/doc/relnotes/mysql-cluster/8.0/en/news-8-0-30.html

MySQL 8.0.30: Release Notes
- https://dev.mysql.com/doc/relnotes/mysql/8.0/en/news-8-0-30.html
- https://dev.mysql.com/doc/relnotes/mysql-shell/8.0/en/news-8-0-30.html
- https://dev.mysql.com/doc/relnotes/mysql-router/en/news-8-0-30.html
- https://dev.mysql.com/doc/relnotes/mysql-operator/en/news-8-0-30.html
- https://dev.mysql.com/doc/relnotes/workbench/en/news-8-0-30.html

MySQL 8.0.30: Connectors Release Notes
- https://dev.mysql.com/doc/relnotes/connector-odbc/en/news-8-0-30.html
- https://dev.mysql.com/doc/relnotes/connector-j/8.0/en/news-8-0-30.html
- https://dev.mysql.com/doc/relnotes/connector-python/en/news-8-0-30.html
- https://dev.mysql.com/doc/relnotes/connector-net/en/news-8-0-30.html
- https://dev.mysql.com/doc/relnotes/connector-nodejs/en/news-8-0-30.html
- https://dev.mysql.com/doc/relnotes/connector-cpp/en/news-8-0-30.html

How to build NDB clusters from source code in arm ubuntu (1 reply)

August 22, 2022, 1:01 am

≫ Next: Insert large longtest failed on NDB (1 reply)

≪ Previous: MySQL NDB Cluster 8.0.30: Release Notes (no replies)

$

0

0

How to build NDB clusters from source code in arm ubuntu, or how to get arm NDB clusters

Insert large longtest failed on NDB (1 reply)

November 3, 2022, 4:55 pm

≫ Next: Node2 & Node3 Cluster requires this node to be started with --initial (2 replies)

≪ Previous: How to build NDB clusters from source code in arm ubuntu (1 reply)

$

0

0

Hello Everyone,

We want to use longtext column to save big data for an application. I got the error below when trying to insert a 500M string into a longtext column on a NDB. The same insert can be done on a SQL node as a standalone InnoDB.

I checked the documents, default value of MaxDMLOperationsPerTransaction is already maximum 4294967295. The NDB is isolated and should not have concurrent operations. I don't know what we can do.

Anyone please help.

Error:
Error Code: 1296. Got error 261 'DML count in transaction exceeds config parameter MaxDMLOperationsPerTransaction/MaxNoOfConcurrentOp' from NDBCLUSTER

Table:
CREATE TABLE `big_data_test` (
`id` int NOT NULL AUTO_INCREMENT,
`val` longtext,
`record_id` int DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=ndbcluster AUTO_INCREMENT=5 DEFAULT CHARSET=ascii;

SQL:
INSERT INTO big_data_test (val, record_id)
VALUES (REPEAT('x',1024*1024*500),3);

NDB:
1 management node 2G RAM
2 data nodes 6G RAM
1 SQL node 4G RAM

[ndbd default]
NoOfReplicas=1
DataMemory=1G
SharedGlobalMemory=1G
FragmentLogFileSize=64M
TimeBetweenLocalCheckpoints=12

Node2 & Node3 Cluster requires this node to be started with --initial (2 replies)

November 5, 2022, 10:26 am

≫ Next: About High availability (1 reply)

≪ Previous: Insert large longtest failed on NDB (1 reply)

$

0

0

Hi all newbie here (well less than a newbie, idiot).

I have a 3 node MySQL cluster that was installed via Docker (MySQL Docker Image 7.5.10-1.1.5), which this deployment was created by someone that isn't available to me anymore. The DB is used for Grafana and each of the Grafana's are operational and the 3 Nodes Grafana DBs are in sync.

The problem is on Node2 (Zoo2) and Node3 (Zoo3) the NDBD keeps restarting. I am pretty sure the logging is telling me what do to with the "Cluster requires this node to be started with --initial" log entry. Though I can figure out what that really means or how to start the docker cluster node with "--initial"... because I am to stupid to figure it out with all the reading I have found on the internet about Docker/MySQL.

I do a apologize for coming in here with "Hat in Hand" and not being able to understand this MySQL Cluster setup myself. I hope not to be flamed to hard and someone will have pity on me.


{"log":"2022-11-05 13:41:43 [ndbd] INFO -- NDB start phase 1 completed\n","stream":"stdout","time":"2022-11-05T13:41:43.252099351Z"}
{"log":"2022-11-05 13:41:43 [ndbd] INFO -- Start phase 2 completed\n","stream":"stdout","time":"2022-11-05T13:41:43.252102151Z"}
{"log":"2022-11-05 13:41:43 [ndbd] INFO -- Phase 2 did more initialisations, master accepted our start, we started REDO log initialisations\n","stream":"stdout","time":"2022-11-05T13:41:43.252104554Z"}
{"log":"2022-11-05 13:41:43 [ndbd] INFO -- Start NDB start phase 2\n","stream":"stdout","time":"2022-11-05T13:41:43.252108664Z"}
{"log":"2022-11-05 13:41:43 [ndbd] INFO -- Request permission to start our node from master Starting\n","stream":"stdout","time":"2022-11-05T13:41:43.275526623Z"}
{"log":"2022-11-05 13:41:43 [ndbd] INFO -- Cluster requires this node to be started with --initial as partial start has been performed and this filesystem is unusable\n","stream":"stdout","time":"2022-11-05T13:41:43.282072895Z"}
{"log":"2022-11-05 13:41:43 [ndbd] INFO -- DBDIH (Line: 2267) 0x00000002\n","stream":"stdout","time":"2022-11-05T13:41:43.282083458Z"}
{"log":"2022-11-05 13:41:43 [ndbd] INFO -- Error handler shutting down system\n","stream":"stdout","time":"2022-11-05T13:41:43.282094445Z"}
{"log":"2022-11-05 13:41:43 [ndbd] INFO -- Error handler shutdown completed - exiting\n","stream":"stdout","time":"2022-11-05T13:41:43.320431813Z"}
{"log":"2022-11-05 13:41:43 [ndbd] ALERT -- Node 5: Forced node shutdown completed. Occured during startphase 3. Caused by error 2311: 'Conflict when selecting restart type(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'.\n","stream":"stdout","time":"2022-11-05T13:41:43.375021266Z"}

f13e0be154f557317a34c6267b0a0847247aa4073775e7f3d46a3720da4b2043 10.46.133.180:5000/mysql/mysql-cluster "/entrypoint.sh mysqld --ndb-nodeid=8" 2 months ago Up 2 hours (healthy) mysqlcluster_mysql2_1
d80a9e1a859d01e3812c40d7802edeba62842906eed91c858c74f95a6d55bca3 10.46.133.180:5000/mysql/mysql-cluster "/entrypoint.sh ndbd --ndb-nodeid=5 --nowait-nodes=5,6" 2 months ago Up 40 seconds (health: starting) mysqlcluster_ndb2_1
724caf3849d69a72c135065c424a06813cff8e565f3dbe7054a91febe7dc08a7 10.46.133.180:5000/mysql/mysql-cluster "/entrypoint.sh ndb_mgmd --ndb-nodeid=2" 2 months ago Up 2 hours (unhealthy) mysqlcluster_management2_1

About High availability (1 reply)

November 21, 2022, 11:24 am

≫ Next: RH/Oracle Linux 9 Yum Repo Empty (1 reply)

≪ Previous: Node2 & Node3 Cluster requires this node to be started with --initial (2 replies)

$

0

0

Hi all, i've installed mysql cluster via deb and apt.
i followed this guide: https://dev.mysql.com/doc/mysql-apt-repo-quick-guide/en/#repo-qg-apt-cluster-install
-----------------------------------------------------
i want to have 2 VM running management and SQL together
and 2 VM for data node on.

So it's possible to have mysqld and ndb_mgmd running on same VM x 2?
i've tried but it's not working.
----------------------------------------------------------
here my /etc/my.cnf on VM1 and VM2:
[mysqld]
ndbcluster

[mysql_cluster]
ndb-connectstring=dbmng1,dbmng2
-----------------------------------------------------------
here my config.ini on VM1 and VM2:
[ndbd default]
NoOfReplicas=2
DataMemory=98M

[ndb_mgmd]
NodeId=1
HostName=dbmng1
DataDir=/var/lib/mysql-cluster

[ndb_mgmd]
NodeId=2
HostName=dbmng2
DataDir=/var/lib/mysql-cluster

[ndbd]
NodeId=3
HostName=dbdata1
DataDir=/usr/local/mysql/data

[ndbd]
NodeId=4
HostName=dbdata2
DataDir=/usr/local/mysql/data

[mysqld]
HostName=dbmng1

[mysqld]
HostName=dbmng2
--------------------------------------------------

RH/Oracle Linux 9 Yum Repo Empty (1 reply)

December 2, 2022, 2:54 am

≫ Next: MySQL Cluster 8 - 4 nodes (1 reply)

≪ Previous: About High availability (1 reply)

$

0

0

I am going to add two more data nodes to my cluster and am in the process of setting up new servers. I am planning to use Oracle Linux 9. However, the yum repository for that version doesn't have any RPMs in it:

http://repo.mysql.com/yum/mysql-cluster-8.0-community/el/9/x86_64/

Does anyone know if this is only temporary or will RPMs eventually be there? I normally use yum to keep my systems up to date.

Thanks!

---

MySQL Cluster 8 - 4 nodes (1 reply)

January 4, 2023, 5:32 am

≫ Next: MySQL NDB Cluster 8.0.31 Transparent Data Encryption (no replies)

≪ Previous: RH/Oracle Linux 9 Yum Repo Empty (1 reply)

$

0

0

Hello.

So we are running 4 nodes and 2 masters

the same exact as this

https://www.mysql.com/products/cluster/availability.html


But if 2 of my nodes lose connection or go down they wont auto reconnect i have tried a few different things, and if i just restart them manually they crash the entire cluster.

and if i restart the remaing 2 nodes they wont start because it says not enough nodes to start...

note: im saying if an A from group 0 and B from group 1 goes down so its still actually running so that the partitions are still accessable


I am not sure in what world this is HA.

Is there a fix for this?

MySQL NDB Cluster 8.0.31 Transparent Data Encryption (no replies)

December 22, 2022, 1:39 pm

≫ Next: NDB 8.0 Example Doesn't Work (1 reply)

≪ Previous: MySQL Cluster 8 - 4 nodes (1 reply)

$

0

0

MySQL NDB Cluster 8.0.31 Transparent Data Encryption
- https://blogs.oracle.com/mysql/post/mysql-cluster-8031-transparent-data-encryption

NDB 8.0 Example Doesn't Work (1 reply)

January 4, 2023, 7:13 am

≫ Next: NDB: stored grants: not intialized. (1 reply)

≪ Previous: MySQL NDB Cluster 8.0.31 Transparent Data Encryption (no replies)

$

0

0

I'm trying to follow this tutorial: https://dev.mysql.com/doc/refman/8.0/en/mysql-cluster-config-starting.html

Attempting to start the second management node always dies. The first node rejects the second.

#First node started logs this: Refusing other node, it has different checksum: 0x24d61568, expected: 0x25d62474

#Second node started logs this:-- Got CONFIG_CHECK_REF from node 1, error: 3, message: 'Wrong checksum', generation: 0, expected generation: 0, state: 1, expected state: 1 own-state: 1

I copy and pasted the config and ONLY changed the host names and NodeId.

It doesn't help if I start with --initial on one or both nodes. Thoughts?

NDB: stored grants: not intialized. (1 reply)

January 4, 2023, 7:55 am

≫ Next: How to restart or recover data when all data nodes crash and then fail to complete start up due to frag log error in phase 4 (1 reply)

≪ Previous: NDB 8.0 Example Doesn't Work (1 reply)

$

0

0

Attempting to get my first cluster going and adding users ran into this message.

NDB: stored grants: not intialized.

I followed these instructions:
https://dev.mysql.com/doc/mysql-cluster-excerpt/8.0/en/mysql-cluster-privilege-synchronization.html

This is in a kubernetes environment and deployed with a bash script on boot. I have "sleep 5" after the server starts and forks away, This is in a file:


GRANT NDB_STORED_USER ON *.* TO `root`@`localhost`

FLUSH PRIVILEGES;

CREATE USER IF NOT EXISTS 'ndb_demo'@'%' IDENTIFIED BY 'suuuuperseeeecret';

Then I get the error and I'm not finding any documentation that explains it.

Should I sleep longer to let the sql server join the cluster?

How to restart or recover data when all data nodes crash and then fail to complete start up due to frag log error in phase 4 (1 reply)

February 17, 2023, 11:20 pm

≫ Next: Error with Indexes (1 reply)

≪ Previous: NDB: stored grants: not intialized. (1 reply)

$

0

0

We've been running an older version of MySQL Cluster 7.4.7 for many years with minimum configuration, containing 2 data nodes, 2 sql nodes, and 1 management node.
After running an Alter Table statement to add 2 big integer columns to a table with 1,000,000 rows. the query failed and crashed the 1st data node. Upon running the query again it crashed the 2nd data node, bringing down the whole cluster.
When trying to restart either data node, they both fail during start up calling in phase 4 when reaching a frag file. see error below in logs.

So the question is, is it possible to repair or have the data nodes ignore the frag file(s) that's stopping the start up process from completing? Or how do you get around this "File has already been opened" error? We tried moving the problematic S11.FragLog file but then errors get reported that a file is missing.
If there is no way to fix these broken frag files, is there a way to recover the table data from a data node (such as using a separate tool to read/export the node's data stored on disk)?
A more general question is how do you recover data when all data nodes in a cluster cannot start (say due a single corrupt file) and there's no separate backups available?


$./ndbmtd
...
2023-02-11 06:05:13 [ndbd] INFO -- LDM instance 0: Restored T1474F1 LCP 0 rows, 1 millis, 0 rows/s)
2023-02-11 06:05:13 [ndbd] INFO -- LDM instance 0: Restored T1477F0 LCP 161953 rows, 622 millis, 260374 rows/s)
2023-02-11 06:05:14 [ndbd] INFO -- LDM instance 0: Restored T1477F1 LCP 163565 rows, 630 millis, 259626 rows/s)
2023-02-11 06:05:14 [ndbd] INFO -- LDM instance 0: Restored T1482F0 LCP 1 rows, 1 millis, 1000 rows/s)
2023-02-11 06:05:14 [ndbd] INFO -- LDM instance 0: Restored T1482F1 LCP 3 rows, 1 millis, 3000 rows/s)
2023-02-11 06:05:14 [ndbd] INFO -- LDM instance 0: Restored LCP : 1114 fragments, 47359940 rows, 160001 millis, 295997 rows/s
2023-02-11 06:05:14 [ndbd] INFO -- LDM(0): Completed fuzzy restore 1118 fragments (1118 from LCP, 0 by non-trans copy)
2023-02-11 06:05:14 [ndbd] INFO -- LDM(0): Starting DD Undo log application
2023-02-11 06:05:14 [ndbd] INFO -- LDM(0): Completed DD Undo log application
2023-02-11 06:05:14 [ndbd] INFO -- LDM(0): Starting REDO log execution phase 0
2023-02-11 06:05:14 [ndbd] INFO -- LDM(0): Node 3 ready to execute REDO log
2023-02-11 06:05:14 [ndbd] INFO -- LDM(0): All starting nodes ready to execute REDO log. Phases completed = 0
2023-02-11 06:05:14 [ndbd] INFO -- LDM(0):Ready to start execute REDO log phase, collect REDO log execution info phase completed
2023-02-11 06:05:14 [ndbd] INFO -- LDM(0): Log part 0 will execute REDO log records from GCI 85012871 -> 85013314 and last log file is number 11 and last MByte in this file is 13
2023-02-11 06:05:14 [ndbd] INFO -- LDM(0): Log part 1 will execute REDO log records from GCI 85012871 -> 85013314 and last log file is number 2 and last MByte in this file is 12
2023-02-11 06:05:14 [ndbd] INFO -- LDM(0): Log part 2 will execute REDO log records from GCI 85012871 -> 85013314 and last log file is number 6 and last MByte in this file is 6
2023-02-11 06:05:14 [ndbd] INFO -- LDM(0): Log part 3 will execute REDO log records from GCI 85012871 -> 85013314 and last log file is number 11 and last MByte in this file is 14
2023-02-11 06:05:14 [ndbd] INFO -- LDM(0): Start executing REDO log for part 0
2023-02-11 06:05:14 [ndbd] INFO -- LDM(0): Start executing REDO log for part 1
2023-02-11 06:05:14 [ndbd] INFO -- LDM(0): Start executing REDO log for part 2
2023-02-11 06:05:14 [ndbd] INFO -- LDM(0): Start executing REDO log for part 3
2023-02-11 06:05:14 [ndbd] INFO -- LDM(0): Completed REDO log execution on part 1, ops executed = 0, bytes executed = 0,
2023-02-11 06:05:14 [ndbd] INFO -- LDM(0): Log part 1 stats: ops skipped = 735, ops prepared 735, pages read = 8, GCIs executed = 444
2023-02-11 06:05:14 [ndbd] INFO -- LDM(0): Completed REDO log execution on part 0, ops executed = 388, bytes executed = 7212,
2023-02-11 06:05:14 [ndbd] INFO -- LDM(0): Log part 0 stats: ops skipped = 639, ops prepared 1027, pages read = 8, GCIs executed = 1283
2023-02-11 06:05:14 [ndbd] INFO -- LDM(0): Completed REDO log execution on part 2, ops executed = 249, bytes executed = 49328,
2023-02-11 06:05:14 [ndbd] INFO -- LDM(0): Log part 2 stats: ops skipped = 7320, ops prepared 7569, pages read = 48, GCIs executed = 2080
2023-02-11 06:05:15 [ndbd] INFO -- open: >/var/lib/mysql-cluster/data/ndb_3_fs/D11/DBLQH/S11.FragLog< existing: >/var/lib/mysql-cluster/data/ndb_3_fs/D11/DBLQH/S11.FragLog<
2023-02-11 06:05:15 [ndbd] INFO -- OpenFiles::insert()
2023-02-11 06:05:15 [ndbd] INFO -- Error handler shutting down system
2023-02-11 06:05:15 [ndbd] INFO -- Error handler shutdown completed - exiting
2023-02-11 06:05:18 [ndbd] ALERT -- Node 3: Forced node shutdown completed. Occured during startphase 4. Caused by error 2807: 'File has already been opened(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'.

Error with Indexes (1 reply)

February 25, 2023, 12:04 pm

≫ Next: need to add new ndb data node to my cluster (1 reply)

≪ Previous: How to restart or recover data when all data nodes crash and then fail to complete start up due to frag log error in phase 4 (1 reply)

$

0

0

I added to more nodes to my cluster. I put them in a separate node group and everything ran fine for a week. Then I noticed that the new nodes didn't have much data so I ran the alter table command to redistribute the data. During that, the cluster crashed. I could not restart it and was forced to restore from backup. Now, however, I get this error:

| xxxx | analyze | Error | Got error 4721 'Mysqld: index stats thread not open for requests' from NDB |
| xxxx | analyze | Warning | index stats (RPK) for key vote_id: unexpected error 4721 |
| xxxx | analyze | Warning | index stats (RPK) for key vote_id: unexpected error 4721 |
| xxxx | analyze | status | Operation failed

Everything is working very slowly so I assume that indexes are not being used. I did not import indexes the data restoration, but did run the rebuild command separately. Does anyone have any idea what to do about this problem?

need to add new ndb data node to my cluster (1 reply)

March 10, 2023, 2:53 am

≫ Next: Mysql NDB Cluster error while restoration of data "ERROR 1114 (HY000) at line 25: The table 'table1' is full" (1 reply)

≪ Previous: Error with Indexes (1 reply)

$

0

0

Dear,
we have ndb cluster 7.4.10 with mysql version is 5.6.28 ndb-7.4.10
we have 2 data nodes with replica 2.
can we add another third node to this cluster?

why when we start the two nodes the commands hungs. for example when i try to use use DB1 it hangs
why it is hungging?

please, advise.

---

Mysql NDB Cluster error while restoration of data "ERROR 1114 (HY000) at line 25: The table 'table1' is full" (1 reply)

May 2, 2023, 6:48 am

≫ Next: XA transaction causes Mysql crash in 8.0.34 (no replies)

≪ Previous: need to add new ndb data node to my cluster (1 reply)

$

0

0

I have created mysql NDB Cluster with two data nodes and two mysql servers.
below is the configuration of ndb_mgmd.

$cat /usr/mysql-cluster/config.ini
[ndbd default]
NoOfReplicas=2
DataMemory=1024M
IndexMemory=54M
MaxNoOfConcurrentOperations=100000
MaxNoOfLocalOperations=110000
MaxNoOfTables=4096
MaxNoOfTriggers=3500
MaxNoOfAttributes=25000

[ndb_mgmd]
hostname=172.31.16.19
datadir=/usr/mysql-cluster
NodeId=1

[ndbd]
hostname=172.31.20.59
NodeId=11
datadir=/usr/local/mysql/data

[ndbd]
hostname=172.31.16.136
NodeId=12
datadir=/usr/local/mysql/data

[mysqld]
hostname=172.31.16.136
NodeId=21

[mysqld]
hostname=172.31.20.59
NodeId=22


When i am trying to restore the databackup .sql file. which is around 2.2 GB in size at that time i am getting error of table is full.
i have checked the disk storage but it is showing available.
Please suggest, i am stuck with the issue from last two days.

XA transaction causes Mysql crash in 8.0.34 (no replies)

August 16, 2023, 1:14 am

≫ Next: NBBCluster not sync autoincrement values between mysql server nodes (2 replies)

≪ Previous: Mysql NDB Cluster error while restoration of data "ERROR 1114 (HY000) at line 25: The table 'table1' is full" (1 reply)

$

0

0

I have the following snippet:

# cat xacrash.sql
drop database if exists newdb;
create database newdb;
use newdb;
create table innodbtable(id int) engine=innodb;
xa begin 'innodbtx';
insert into innodbtable values(3);
xa end 'innodbtx';
xa prepare 'innodbtx';
xa commit 'innodbtx';
create table ndbtable(id int) engine=ndbcluster;
xa begin 'ndbtx';
insert into ndbtable values(3);
xa end 'ndbtx';
xa prepare 'ndbtx';
xa commit 'ndbtx';

When running against 8.0.28 works without problems, but crashes connection in 8.0.34

--------------
xa prepare 'ndbtx'
--------------

ERROR 2013 (HY000) at line 14: Lost connection to MySQL server during query

And writes a crash report

Aug 16 10:08:54 hostname mysqld[606104]: 2023-08-16T08:08:54Z UTC - mysqld got signal 11 ;
Aug 16 10:08:54 hostname mysqld[606104]: Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
Aug 16 10:08:54 hostname mysqld[606104]: BuildID[sha1]=a570f2d544031bf853200e0bb758215b0b317fb1
Aug 16 10:08:54 hostname mysqld[606104]: Thread pointer: 0x7fa758000fe0
Aug 16 10:08:54 hostname mysqld[606104]: Attempting backtrace. You can use the following information to find out
Aug 16 10:08:54 hostname mysqld[606104]: where mysqld died. If you see no messages after this, something went
Aug 16 10:08:54 hostname mysqld[606104]: terribly wrong...
Aug 16 10:08:54 hostname mysqld[606104]: stack_bottom = 7fa7b42edc10 thread_stack 0x100000
Aug 16 10:08:54 hostname mysqld[606104]: /usr/sbin/mysqld(my_print_stacktrace(unsigned char const*, unsigned long)+0x41) [0x2131091]
Aug 16 10:08:54 hostname mysqld[606104]: /usr/sbin/mysqld(print_fatal_signal(int)+0x2a2) [0xfee7d2]
Aug 16 10:08:54 hostname mysqld[606104]: /usr/sbin/mysqld(handle_fatal_signal+0xa5) [0xfee985]
Aug 16 10:08:54 hostname mysqld[606104]: /lib64/libpthread.so.0(+0x12cf0) [0x7fa7dcabbcf0]
Aug 16 10:08:54 hostname mysqld[606104]: Trying to get some variables.
Aug 16 10:08:54 hostname mysqld[606104]: Some pointers may be invalid and cause the dump to abort.
Aug 16 10:08:54 hostname mysqld[606104]: Query (7fa7581af9c0): xa prepare 'ndbtx'
Aug 16 10:08:54 hostname mysqld[606104]: Connection ID (thread ID): 14
Aug 16 10:08:54 hostname mysqld[606104]: Status: NOT_KILLED
Aug 16 10:08:54 hostname mysqld[606104]: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
Aug 16 10:08:54 hostname mysqld[606104]: information that should help you find out what is causing the crash.


# rpm -qi mysql-cluster-community-server
Name : mysql-cluster-community-server
Version : 8.0.34
Release : 1.el8
Architecture: x86_64

NBBCluster not sync autoincrement values between mysql server nodes (2 replies)

September 17, 2023, 7:24 am

≫ Next: Error while trying ndb_restore (6 replies)

≪ Previous: XA transaction causes Mysql crash in 8.0.34 (no replies)

$

0

0

I have a NDBcluster on 2 physical servers with this layout:
vm1 ndb manager + mysql server #1
vm2 mysql server #2
vm3 data node #1
vm4 data node #2

i import many dbase on vm1 using mysql commnand line.
i can see and access all data from both vm1 and vm2.
but after 2 months of perfect usage,
vm2 mysql server if i insert data in any table of any dbase start to use
weird and not syncronized value for autoincrement of any id primary key.

example:
on vm1: insert into user (name,email) values ("tony","t@t.com")
insert into user (name,email) values ("john","f@f.com")
result on both vm1 and vm2:
id=8 , name=tony, email=t@t.com
id=9 , name=john, email=f@f.com

then i do on vm2:
insert into user (name,email) values ("kris","k@k.com")
result on both vm1 and vm2:
id=8 , name=tony, email=t@t.com
id=9 , name=john, email=f@f.com
id=521 , name=kris, email=k@k.com

Can someone explain me why this happen? why after 2 months and how to fix this please!

Error while trying ndb_restore (6 replies)

November 29, 2023, 12:57 am

≫ Next: MySQL Cluster (NDB) spread data (rows) to 4 data-nodes servers (no replies)

≪ Previous: NBBCluster not sync autoincrement values between mysql server nodes (2 replies)

$

0

0

Hello,
I am a student and still unfamilliar with mysql cluster. I am trying to do backup and restore.

1. I have config in config.ini for back up as followed:

## BackUp setting
## BackupDataDir automatically follow DataDir /home/user/BACKUP
BackupLogBufferSize=16M
BackupReportFrequency=10

2. This is the cluster that has been configured.

ndb_mgm> show
Cluster Configuration
---------------------
[ndbd(NDB)] 2 node(s)
id=2 @192.168.1.236 (mysql-8.0.35 ndb-8.0.35, Nodegroup: 0, *)
id=3 @192.168.1.237 (mysql-8.0.35 ndb-8.0.35, Nodegroup: 0)

[ndb_mgmd(MGM)] 1 node(s)
id=1 @192.168.1.238 (mysql-8.0.35 ndb-8.0.35)

[mysqld(API)] 3 node(s)
id=10 @192.168.1.239 (mysql-8.0.35 ndb-8.0.35)
id=11 @192.168.1.240 (mysql-8.0.35 ndb-8.0.35)
id=12 @192.168.1.243 (mysql-8.0.35 ndb-8.0.35)


3. I have done backup at ndb_mgm

ndb_mgm > start backup
Connected to Management Server at: localhost:1186
Waiting for completed, this may take several minutes
Node 3: Backup 1 started from node 1
Node 2: Local backup status: backup 1 started from node 1
#Records: 14001933 #LogRecords: 243288
Data: 832059592 bytes Log: 21435756 bytes
.
.
.
Node 3: Backup 1 started from node 1 completed
StartGCP: 201318 StopGCP: 201401
#Records: 28027692 #LogRecords: 656042
Data: 1665876984 bytes Log: 55078372 bytes

4. then, I try deleting some tables.

5. after that, I want to try restoring it. However there are some error, that I can't find the solution.

######First error when I shutdown ndb cluster, then only activate ndb_mgmd.

datanode236:/home/user/# ndb_restore -n 2 -b 1 -m --ndb-connectstring=192.168.1.238:1186 --ndb-nodeid=1 --backup-path=BACKUP/BACKUP-1
Nodeid = 2
Backup Id = 1
backup path = BACKUP/BACKUP-1
2023-11-28 08:37:48 [NdbApi] INFO -- Configuration error: Error: Could not alloc node id at 192.168.1.238:1186: Id 1 configured as ndb_mgmd(MGM), connect attempted as mysqld(API).
Could not connect to cluster: '1 - Configuration error: Error: Could not alloc node id at 192.168.1.238:1186: Id 1 configured as ndb_mgmd(MGM), connect attempted as mysqld(API).'


########Second Error when ndb cluster all running. At datanode 192.168.1.236:

datanode236:/home/user/#ndb_restore -n 2 -b 1 -m --ndb-connectstring=192.168.1.238:1186 --backup-path=BACKUP/BACKUP-1
Nodeid = 2
Backup Id = 1
backup path = BACKUP/BACKUP-1
2023-11-28 08:38:06 [NdbApi] INFO -- Configuration error: Error: Could not alloc node id at 192.168.1.238:1186: Connection done from wrong host ip 192.168.1.236.
Could not connect to cluster: '1 - Configuration error: Error: Could not alloc node id at 192.168.1.238:1186: Connection done from wrong host ip 192.168.1.236.'

Please help if you know what I miss.
Thank you.

MySQL Cluster (NDB) spread data (rows) to 4 data-nodes servers (no replies)

December 2, 2023, 6:12 am

≫ Next: Error 4009 (no replies)

≪ Previous: Error while trying ndb_restore (6 replies)

$

0

0

I want to spread my MySQL rows to 4 servers (Horizontal sharding). I searched, and I found that I have to use NDB cluster (MySQL cluster). I made a MySQL server/client with NDB manager server and 4 data nodes servers and this is my manager configuration :

[ndbd default]
NoOfReplicas=1

[ndb_mgmd]
NodeId=1
hostname=192.168.159.133
datadir=/var/lib/mysql-cluster

[ndbd]
NodeId=2
hostname=192.168.159.131
datadir=/home/db

[ndbd]
NodeId=3
hostname=192.168.159.132
datadir=/home/db

[ndbd]
NodeId=4
hostname=192.168.159.137
datadir=/home/db

[mysqld]
NodeId=20
hostname=192.168.159.133

I know replication means make a backup of each row to another server and I don't want it (I just want to spread my data (rows) to 4 servers, not making backup) so I set NoOfReplicas to 1 to have 4 nodegroups ...

Am I right ? Does my data (rows) spread on that 4 data-node servers ?

If I right, what is different between doing this, or just using Innodb and make foreign partitions (on remote servers) ? Because it's a lot easier to make partition (key) by my own and attach each partition to remote servers !!!

Is using NDB has more advantage (about what I want to do (spread data, not replication (making backup))) ?

And final question, is it right to have NDBCLUSTER table with also own partitions ???!!! What is that mean, anyway ? For example, I have 4 data-nodes and for a NDBCLUSTER table, I create 8 partitions too (key partition) ! How that's work? Does it create 8 partitions on each data-node ? Or just split those partitions on those 4 data-nodes server (for example, each data-node holds 2 partitions) ?

Error 4009 (no replies)

January 5, 2024, 7:35 pm

≫ Next: Error 1064 Duplicate entry 'ndbcluster-96' for key 'tables.engine' (no replies)

≪ Previous: MySQL Cluster (NDB) spread data (rows) to 4 data-nodes servers (no replies)

$

0

0

Hello,

I am making a project using 8 data nodes, and 47 sql nodes. Here's the config.ini setting in ndb management server:

[MGM]
DataDir=/usr/clustergo
HostName=192.168.1.1

[DB default]
DataDir=/usr/local/mysql/cluster
NoOfReplicas=2

## MEMORY MANAGEMENT
DataMemory=200G
StringMemory=25

## TRANSACTION
MaxNoOfConcurrentTransactions=4000000
MaxNoOfConcurrentOperations=8000000

## TRANSACTION RESOURCE ALLOCATION
TransactionMemory=20G

## LOGGING AND CHECK POINTS
FragmentLogFileSize=32M
NoOfFragmentLogFiles=600
EnableRedoControl=1

## METADATA OBJECTS: it defines pool sizes for metadata objects.
MaxNoOfTables=10240
MaxNoOfOrderedIndexes=2560
MaxNoOfUniqueHashIndexes=2560
MaxNoOfAttributes=1200000
MaxNoOfTriggers=2560
MaxNoOfConcurrentSubOperations=1024

## BOOLEAN PARAMETERS
CompressedLCP=1
LockPagesInMainMemory=1
ODirect=1

## CONTROLLING TIMEOUT, INTERVALS AND DISK PAGING
TimeBetweenInactiveTransactionAbortCheck=4000
TransactionDeadlockDetectionTimeout=12000

## Buffering and logging
RedoBuffer=128M

## CONTROLLING LOG MESSAGE
LogLevelStartup=1
LogLevelShutdown=1
LogLevelCheckpoint=8
LogLevelNodeRestart=15
LogLevelConnection=8
LogLevelError=8
LogLevelCongestion=1
MemReportFrequency=30

## BACKUP CONFIG
BackupMaxWriteSize=10M
BackupDataBufferSize=26M
BackupLogBufferSize=32M
BackupReportFrequency=10

## MULTITHREADING CONFIG
AutomaticThreadConfig=1

## PARAMETER FOR SEND BUFFER MEMORY ALLOCATION

TotalSendBufferMemory=128M

##DISK DATA BUFFERING PARAMETER
DiskPageBufferEntries=100
DiskPageBufferMemory=1024M
SharedGlobalMemory=160M


## NDB CLUSTER REALTIME PERFORMANCE
SpinMethod=DatabaseMachineSpinning
SchedulerExecutionTimer=100

[DB]
HostName=192.168.1.2
LockExecuteThreadToCPU=1
LockMaintThreadsToCPU=0

[DB]
HostName=192.168.1.3
LockExecuteThreadToCPU=1
LockMaintThreadsToCPU=0
.
. ## and so on
.
[API]
NodeId=46

[API]
NodeId=47
HostName=192.168.1.30

###

And whenever there is a large request. It got error 4009, like API (sqlnode) suddenly cannot connected to DB (datanode).

Here's log from ndb_1_cluster.log (ndb management):
2024-01-05 16:50:16 [MgmtSrvr] INFO -- Node 4: Index usage is 0%(20523 32K pages of total 5735985)
2024-01-05 16:50:19 [MgmtSrvr] INFO -- Node 3: Data usage is 12%(816695 32K pages of total 6533144)
2024-01-05 16:50:19 [MgmtSrvr] INFO -- Node 3: Index usage is 0%(20456 32K pages of total 5736905)
2024-01-05 16:50:30 [MgmtSrvr] ALERT -- Node 9: Node 47 Disconnected
2024-01-05 16:50:30 [MgmtSrvr] INFO -- Node 9: Communication to Node 47 closed
2024-01-05 16:50:30 [MgmtSrvr] INFO -- Node 7: Communication to Node 47 closed
2024-01-05 16:50:30 [MgmtSrvr] INFO -- Node 3: Communication to Node 47 closed
2024-01-05 16:50:30 [MgmtSrvr] INFO -- Node 6: Communication to Node 47 closed
2024-01-05 16:50:30 [MgmtSrvr] INFO -- Node 4: Communication to Node 47 closed
2024-01-05 16:50:30 [MgmtSrvr] INFO -- Node 5: Communication to Node 47 closed
2024-01-05 16:50:30 [MgmtSrvr] INFO -- Node 2: Communication to Node 47 closed
2024-01-05 16:50:30 [MgmtSrvr] ALERT -- Node 2: Node 47 Disconnected
2024-01-05 16:50:30 [MgmtSrvr] INFO -- Node 8: Communication to Node 47 closed
2024-01-05 16:50:30 [MgmtSrvr] ALERT -- Node 6: Node 47 Disconnected
2024-01-05 16:50:30 [MgmtSrvr] ALERT -- Node 7: Node 47 Disconnected
2024-01-05 16:50:30 [MgmtSrvr] ALERT -- Node 4: Node 47 Disconnected
2024-01-05 16:50:30 [MgmtSrvr] ALERT -- Node 5: Node 47 Disconnected
2024-01-05 16:50:30 [MgmtSrvr] ALERT -- Node 8: Node 47 Disconnected
2024-01-05 16:50:30 [MgmtSrvr] ALERT -- Node 3: Node 47 Disconnected
2024-01-05 16:50:31 [MgmtSrvr] INFO -- Node 6: Data usage is 12%(821829 32K pages of total 6532706)
2024-01-05 16:50:31 [MgmtSrvr] INFO -- Node 6: Index usage is 0%(20894 32K pages of total 5731771)
2024-01-05 16:50:33 [MgmtSrvr] INFO -- Node 6: Communication to Node 47 opened
2024-01-05 16:50:33 [MgmtSrvr] INFO -- Node 3: Communication to Node 47 opened
2024-01-05 16:50:33 [MgmtSrvr] INFO -- Node 8: Communication to Node 47 opened
2024-01-05 16:50:33 [MgmtSrvr] INFO -- Node 5: Communication to Node 47 opened
2024-01-05 16:50:33 [MgmtSrvr] INFO -- Node 9: Communication to Node 47 opened
2024-01-05 16:50:33 [MgmtSrvr] INFO -- Node 7: Communication to Node 47 opened
2024-01-05 16:50:33 [MgmtSrvr] INFO -- Node 2: Communication to Node 47 opened
2024-01-05 16:50:34 [MgmtSrvr] INFO -- Node 4: Communication to Node 47 opened
2024-01-05 16:50:34 [MgmtSrvr] INFO -- Node 9: Node 47 Connected
2024-01-05 16:50:34 [MgmtSrvr] INFO -- Node 6: Node 47 Connected
2024-01-05 16:50:34 [MgmtSrvr] INFO -- Node 8: Node 47 Connected
2024-01-05 16:50:34 [MgmtSrvr] INFO -- Node 9: Node 47: API mysql-8.0.35 ndb-8.0.35
2024-01-05 16:50:34 [MgmtSrvr] INFO -- Node 8: Node 47: API mysql-8.0.35 ndb-8.0.35
2024-01-05 16:50:34 [MgmtSrvr] INFO -- Node 6: Node 47: API mysql-8.0.35 ndb-8.0.35
2024-01-05 16:50:34 [MgmtSrvr] INFO -- Node 2: Node 47 Connected
2024-01-05 16:50:34 [MgmtSrvr] INFO -- Node 7: Node 47 Connected
2024-01-05 16:50:34 [MgmtSrvr] INFO -- Node 7: Node 47: API mysql-8.0.35 ndb-8.0.35
2024-01-05 16:50:34 [MgmtSrvr] INFO -- Node 2: Node 47: API mysql-8.0.35 ndb-8.0.35
2024-01-05 16:50:35 [MgmtSrvr] INFO -- Node 3: Node 47 Connected
2024-01-05 16:50:35 [MgmtSrvr] INFO -- Node 4: Node 47 Connected
2024-01-05 16:50:35 [MgmtSrvr] INFO -- Node 5: Node 47 Connected
2024-01-05 16:50:35 [MgmtSrvr] INFO -- Node 4: Node 47: API mysql-8.0.35 ndb-8.0.35
2024-01-05 16:50:35 [MgmtSrvr] INFO -- Node 3: Node 47: API mysql-8.0.35 ndb-8.0.35
2024-01-05 16:50:35 [MgmtSrvr] INFO -- Node 5: Node 47: API mysql-8.0.35 ndb-8.0.35


So my guess there is something wrong with the connection between api and db.
Although they are in the same block IP, and there is no firewall set-up in between.
I have checked that Ubuntu 20 do not have SELinux, and UFW status is inactive. I also do not set-up any IP tables.

and for the apps, we are using web application, which although in different block IP (192.168.2.xx), It able to ping under 1ms. there is no firewall, SElinux, ufw, or iptables as well.

but still for a long query it can takes 4 sec to 4 minutes to complete if the traffic is not that much crowded.

If it is in high time access, users often found error 4009, which makes another error code 1296, 1297, or even mysql has gone away.
I have been working on this for 2 days, and I can't find the cause.
please help, thank you.

Error 1064 Duplicate entry 'ndbcluster-96' for key 'tables.engine' (no replies)

January 18, 2024, 1:38 pm

≫ Next: Cluster failure (no replies)

≪ Previous: Error 4009 (no replies)

$

0

0

I have a production 8.0.33-cluster MySQL Cluster Community Server.
it works fine in the last 6 months until today.
There are 28 working databases in it.

Since today I can't create any table in any dbase because it show always this error:
Error 1064 Duplicate entry 'ndbcluster-96' for key 'tables.engine'

doesn't matter the db name or table name, the errori is always the same.
For sure there is no duplicate vlues cause i used a new empty database with no tables into it.

Any help please cause online i tried anything in the last 12 hrs but nothing works.
I suppose cluster is not sync or an engine problem.
I can not reinstall all by 0 it's a production server.
Waiting your help.
Willing to pay if someone wants to assist me through my share screen or anydesk.
Thanks

---

Cluster failure (no replies)

November 1, 2021, 8:10 pm

≪ Previous: Error 1064 Duplicate entry 'ndbcluster-96' for key 'tables.engine' (no replies)

$

0

0

Dear,
I was doing a deletion of about 180 thousand records when got the cluster failure,

Is it because of the lack of hardware resource or I can do some configuration to prevent it from happening again?
Please help, thank you!

Here is the failure log:
###
2021-10-22 00:35:39 [ndbd] WARNING -- Watchdog: Warning overslept 471 ms, expected 100 ms.
2021-10-22 00:35:39 [ndbd] INFO -- timerHandlingLab, expected 10ms sleep, not scheduled for: 376 (ms), exec_time 12 us, sys_time 0 us
2021-10-22 17:10:33 [ndbd] INFO -- Node 2 disconnected in recv with errnum: 104 in state: 0
2021-10-22 17:10:33 [ndbd] INFO -- findNeighbours from: 5950 old (left: 2 right: 2) new (65535 65535)
2021-10-22 17:10:33 [ndbd] ALERT -- Network partitioning - arbitration required
2021-10-22 17:10:33 [ndbd] INFO -- President restarts arbitration thread [state=7]
2021-10-22 17:10:33 [ndbd] ALERT -- Arbitration won - positive reply from node 1
2021-10-22 17:10:33 [ndbd] INFO -- NR Status: node=2,OLD=Initial state,NEW=Node failed, fail handling ongoing
2021-10-22 17:10:33 [ndbd] INFO -- Master takeover started from 2
2021-10-22 17:10:33 [ndbd] INFO -- DBTC 0: Started failure handling for node 2
2021-10-22 17:10:33 [ndbd] INFO -- DBTC 0: Starting take over of node 2
2021-10-22 17:10:33 [ndbd] INFO -- DBTC 0: Step NF_CHECK_SCAN completed, failure handling for node 2 waiting for NF_TAKEOVER, NF_CHECK_TRANSACTION, NF_BLOCK_HANDLE.
2021-10-22 17:10:33 [ndbd] INFO -- DBTC 0: Step NF_BLOCK_HANDLE completed, failure handling for node 2 waiting for NF_TAKEOVER, NF_CHECK_TRANSACTION.
2021-10-22 17:10:33 [ndbd] INFO -- DBTC 0: GCP completion 30633610/1 waiting for node failure handling (1) to complete. Seizing record for GCP.
start_resend(0, 2021-10-22 17:10:33 [ndbd] INFO -- DBTC 0: Step NF_CHECK_TRANSACTION completed, failure handling for node 2 waiting for NF_TAKEOVER.
empty bucket (30633610/1 30633610/0) -> active
2021-10-22 17:10:33 [ndbd] INFO -- Started arbitrator node 1 [ticket=be200068e2bfb284]
2021-10-22 17:10:33 [ndbd] INFO -- Adjusting disk write speed bounds due to : Node restart ongoing
2021-10-22 17:10:35 [ndbd] INFO -- DBTC 0: Completed take over of failed node 2
2021-10-22 17:10:35 [ndbd] INFO -- DBTC 0: Step NF_TAKEOVER completed, failure handling for node 2 complete.
2021-10-22 17:10:35 [ndbd] INFO -- DBTC 0: Completing GCP 30633610/1 on node failure takeover completion.
2021-10-22 17:10:35 [ndbd] INFO -- NR Status: node=2,OLD=Node failed, fail handling ongoing,NEW=Node failure handling complete
2021-10-22 17:10:35 [ndbd] INFO -- Node 2 has completed node fail handling
2021-10-22 17:10:36 [ndbd] INFO -- Adjusting disk write speed bounds due to : Node restart finished
job buffer full
Dumping non-empty job queues:
job buffer 0 --> 2, used 31 FULL!

job buffer full
Dumping non-empty job queues:
job buffer 0 --> 2, used 31 FULL!

For help with below stacktrace consult:
https://dev.mysql.com/doc/refman/en/using-stack-trace.html
Also note that stack_bottom and thread_stack will always show up as zero.
2021-10-22 17:14:11 [ndbd] INFO -- Received signal 6. Running error handler.
stack_bottom = 0 thread_stack 0x0
ndbmtd(my_print_stacktrace(unsigned char const*, unsigned long)+0x3d) [0x89fc7d]
ndbmtd(ndb_print_stacktrace()+0x52) [0x84a0e2]
ndbmtd(handler_error+0xab) [0x4f04ab]
/lib64/libc.so.6(+0x36400) [0x7fb433e6a400]
/lib64/libc.so.6(gsignal+0x37) [0x7fb433e6a387]
/lib64/libc.so.6(abort+0x148) [0x7fb433e6ba78]
ndbmtd() [0x87041c]
ndbmtd() [0x874f7e]
ndbmtd(SimulatedBlock::sendSignal(unsigned int, unsigned short, Signal*, unsigned int, JobBufferLevel) const+0x195) [0x866915]
ndbmtd(Dbtc::releaseAndAbort(Signal*, Dbtc::ApiConnectRecord*)+0x144) [0x6821f4]
ndbmtd(Dbtc::abort015Lab(Signal*, Ptr<Dbtc::ApiConnectRecord>)+0x299) [0x695e69]
ndbmtd(Dbtc::execCONTINUEB(Signal*)+0x931) [0x6b92b1]
ndbmtd() [0x870c68]
ndbmtd() [0x875413]
ndbmtd(mt_job_thread_main+0x249) [0x87a009]
ndbmtd() [0x848308]
/lib64/libpthread.so.0(+0x7ea5) [0x7fb43550aea5]
/lib64/libc.so.6(clone+0x6d) [0x7fb433f328dd]
For help with below stacktrace consult:
https://dev.mysql.com/doc/refman/en/using-stack-trace.html
Also note that stack_bottom and thread_stack will always show up as zero.
stack_bottom = 0 thread_stack 0x0
ndbmtd(my_print_stacktrace(unsigned char const*, unsigned long)+0x3d) [0x89fc7d]
ndbmtd(ndb_print_stacktrace()+0x52) [0x84a0e2]
ndbmtd(ErrorReporter::handleError(int, char const*, char const*, NdbShutdownType)+0x2f) [0x80678f]
ndbmtd(handler_error+0x100) [0x4f0500]
/lib64/libc.so.6(+0x36400) [0x7fb433e6a400]
/lib64/libc.so.6(gsignal+0x37) [0x7fb433e6a387]
/lib64/libc.so.6(abort+0x148) [0x7fb433e6ba78]
ndbmtd() [0x87041c]
ndbmtd() [0x874f7e]
ndbmtd(SimulatedBlock::sendSignal(unsigned int, unsigned short, Signal*, unsigned int, JobBufferLevel) const+0x195) [0x866915]
ndbmtd(Dbtc::releaseAndAbort(Signal*, Dbtc::ApiConnectRecord*)+0x144) [0x6821f4]
ndbmtd(Dbtc::abort015Lab(Signal*, Ptr<Dbtc::ApiConnectRecord>)+0x299) [0x695e69]
ndbmtd(Dbtc::execCONTINUEB(Signal*)+0x931) [0x6b92b1]
ndbmtd() [0x870c68]
ndbmtd() [0x875413]
ndbmtd(mt_job_thread_main+0x249) [0x87a009]
ndbmtd() [0x848308]
/lib64/libpthread.so.0(+0x7ea5) [0x7fb43550aea5]
/lib64/libc.so.6(clone+0x6d) [0x7fb433f328dd]
2021-10-22 17:14:11 [ndbd] INFO -- Signal 6 received; Aborted
2021-10-22 17:14:11 [ndbd] INFO -- /export/home2/pb2/build/sb_1-39758149-1592609781.29/rpm/BUILD/mysql-cluster-gpl-8.0.21/mysql-cluster-gpl-8.0.21/storage/ndb/src/kernel/ndbd.cpp
2021-10-22 17:14:11 [ndbd] INFO -- Error handler signal shutting down system
2021-10-22 17:14:11 [ndbd] INFO -- Error handler shutdown completed - exiting
2021-10-22 17:14:11 [ndbd] ALERT -- Node 3: Forced node shutdown completed. Initiated by signal 6. Caused by error 6000: 'Error OS signal received(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'.
2021-10-22 19:22:59 [ndbd] INFO -- Angel pid: 26660 started child: 26661
###