Quantcast
Channel: MySQL Forums - NDB clusters
Viewing all articles
Browse latest Browse all 1562

NDB Restore freeze (no replies)

$
0
0
I have configured a ndb database cluster with the following nodes:
-4 x Data Nodes (6CPU, 16GB RAM, Ubuntu 20.04)
-2 x SQL Nodes (4 CPU, 8GB RAM, Ubuntu 20.04)
-2 x Management Nodes (2 CPU, 4GB RAM, Ubuntu 20.04)
-2 x LoadBalancer with a Virtual IP for external access

Cluster is working correctly but I can't manage to perform a full restore using ndb_restore. Database contains milions of data. Dump size is around 8GB.
When I start a restore, process freezes and it gives no error. I tried to wait but nothing happened. I analyzed logs but there's nothing in particular about that. SQL Node drops its ID and connection and then nothing happens.

Every time to restart the cluster I have to shutdown datanodes and restart with --initial reimporting all data.

This is the procedure I follow:
ndb_restore --ndb_connectstring=192.168.1.1,192.168.1.2 --nodeid=100 --disable-indexes --restore_meta --backupid=5 --backup_path="BACKUP PATH" --verbose=255
(Only one time to import schema)
ndb_restore --ndb_connectstring=192.168.1.1,192.168.1.2 --nodeid=100 --disable-indexes --restore_data --backupid=5 --backup_path="BACKUP_PATH" --verbose=255
(One for each dataNode 100,101,102,103)

Schema restores succesfully but sometimes data restore freezes. It's strange becuase it freezes at different times and points.

This is the configuration:
SharedGlobalMemory=1G
MaxNoOfExecutionThreads=4
MaxNoOfConcurrentOperations=1000000
MaxNoOfTables=4096
TransactionDeadlockDetectionTimeout=100000000
TransactionInactiveTimeout=100000000
MaxNoOfTriggers=4096 # Default is 768
FragmentLogFileSize=512M
NoOfFragmentLogFiles=100
RealTimeScheduler=1
MinDiskWriteSpeed=10M
MaxDiskWriteSpeed=50M


[ndb_mgmd]
HostName=192.168.1.1
NodeId=1
DataDir=/var/lib/mysql-cluster

[ndb_mgmd]
HostName=192.168.1.2
NodeId=2
DataDir=/var/lib/mysql-cluster

[ndbd]
HostName=192.168.1.200
NodeId=100
DataDir=/usr/local/mysql/data

[ndbd]
HostName=192.168.1.201
NodeId=101
DataDir=/usr/local/mysql/data

[ndbd]
HostName=192.168.1.202
NodeId=102
DataDir=/usr/local/mysql/data

[ndbd]
HostName=192.168.1.203
NodeId=103
DataDir=/usr/local/mysql/data

[mysqld]
NodeId=50
HostName=192.168.1.100

[mysqld]
NodeId=51
HostName=192.168.1.101

Process stacks at points like this:
Processing data in table: DBNAME/def/TABLENAME(211) fragment 0

When I start a restore and it freezes this is the log result in the dataNode
DataNode log:
2021-02-17 12:24:27 [ndbd] INFO -- Node 50 disconnected in state: 0
2021-02-17 12:25:32 [ndbd] INFO -- Node 50 disconnected in recv with errnum: 104 in state: 0
2021-02-17 12:26:20 [ndbd] INFO -- Node 50 disconnected in state: 0
2021-02-17 12:27:23 [ndbd] INFO -- Node 50 disconnected in state: 0
2021-02-17 12:28:50 [ndbd] INFO -- Node 50 disconnected in state: 0
2021-02-17 12:29:25 [ndbd] INFO -- Node 50 disconnected in state: 0
2021-02-17 12:30:08 [ndbd] INFO -- Node 50 disconnected in recv with errnum: 104 in state: 0
2021-02-17 12:30:34 [ndbd] INFO -- part: 1 : time to complete: 1
2021-02-17 12:30:39 [ndbd] INFO -- Node 50 disconnected in state: 0
2021-02-17 12:33:00 [ndbd] INFO -- Node 50 disconnected in state: 0

Any help would be greatly appreciated, thanks a lot!

Giuseppe Rossi

Viewing all articles
Browse latest Browse all 1562

Trending Articles



<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>