I have configured a ndb database cluster with the following nodes:
-4 x Data Nodes (6CPU, 16GB RAM, Ubuntu 20.04)
-2 x SQL Nodes (4 CPU, 8GB RAM, Ubuntu 20.04)
-2 x Management Nodes (2 CPU, 4GB RAM, Ubuntu 20.04)
-2 x LoadBalancer with a Virtual IP for external access
Cluster is working correctly but I can't manage to perform a full restore using ndb_restore. Database contains milions of data. Dump size is around 8GB.
When I start a restore, process freezes and it gives no error. I tried to wait but nothing happened. I analyzed logs but there's nothing in particular about that. SQL Node drops its ID and connection and then nothing happens.
Every time to restart the cluster I have to shutdown datanodes and restart with --initial reimporting all data.
This is the procedure I follow:
ndb_restore --ndb_connectstring=192.168.1.1,192.168.1.2 --nodeid=100 --disable-indexes --restore_meta --backupid=5 --backup_path="BACKUP PATH" --verbose=255
(Only one time to import schema)
ndb_restore --ndb_connectstring=192.168.1.1,192.168.1.2 --nodeid=100 --disable-indexes --restore_data --backupid=5 --backup_path="BACKUP_PATH" --verbose=255
(One for each dataNode 100,101,102,103)
Schema restores succesfully but sometimes data restore freezes. It's strange becuase it freezes at different times and points.
This is the configuration:
SharedGlobalMemory=1G
MaxNoOfExecutionThreads=4
MaxNoOfConcurrentOperations=1000000
MaxNoOfTables=4096
TransactionDeadlockDetectionTimeout=100000000
TransactionInactiveTimeout=100000000
MaxNoOfTriggers=4096 # Default is 768
FragmentLogFileSize=512M
NoOfFragmentLogFiles=100
RealTimeScheduler=1
MinDiskWriteSpeed=10M
MaxDiskWriteSpeed=50M
[ndb_mgmd]
HostName=192.168.1.1
NodeId=1
DataDir=/var/lib/mysql-cluster
[ndb_mgmd]
HostName=192.168.1.2
NodeId=2
DataDir=/var/lib/mysql-cluster
[ndbd]
HostName=192.168.1.200
NodeId=100
DataDir=/usr/local/mysql/data
[ndbd]
HostName=192.168.1.201
NodeId=101
DataDir=/usr/local/mysql/data
[ndbd]
HostName=192.168.1.202
NodeId=102
DataDir=/usr/local/mysql/data
[ndbd]
HostName=192.168.1.203
NodeId=103
DataDir=/usr/local/mysql/data
[mysqld]
NodeId=50
HostName=192.168.1.100
[mysqld]
NodeId=51
HostName=192.168.1.101
Process stacks at points like this:
Processing data in table: DBNAME/def/TABLENAME(211) fragment 0
When I start a restore and it freezes this is the log result in the dataNode
DataNode log:
2021-02-17 12:24:27 [ndbd] INFO -- Node 50 disconnected in state: 0
2021-02-17 12:25:32 [ndbd] INFO -- Node 50 disconnected in recv with errnum: 104 in state: 0
2021-02-17 12:26:20 [ndbd] INFO -- Node 50 disconnected in state: 0
2021-02-17 12:27:23 [ndbd] INFO -- Node 50 disconnected in state: 0
2021-02-17 12:28:50 [ndbd] INFO -- Node 50 disconnected in state: 0
2021-02-17 12:29:25 [ndbd] INFO -- Node 50 disconnected in state: 0
2021-02-17 12:30:08 [ndbd] INFO -- Node 50 disconnected in recv with errnum: 104 in state: 0
2021-02-17 12:30:34 [ndbd] INFO -- part: 1 : time to complete: 1
2021-02-17 12:30:39 [ndbd] INFO -- Node 50 disconnected in state: 0
2021-02-17 12:33:00 [ndbd] INFO -- Node 50 disconnected in state: 0
Any help would be greatly appreciated, thanks a lot!
Giuseppe Rossi
-4 x Data Nodes (6CPU, 16GB RAM, Ubuntu 20.04)
-2 x SQL Nodes (4 CPU, 8GB RAM, Ubuntu 20.04)
-2 x Management Nodes (2 CPU, 4GB RAM, Ubuntu 20.04)
-2 x LoadBalancer with a Virtual IP for external access
Cluster is working correctly but I can't manage to perform a full restore using ndb_restore. Database contains milions of data. Dump size is around 8GB.
When I start a restore, process freezes and it gives no error. I tried to wait but nothing happened. I analyzed logs but there's nothing in particular about that. SQL Node drops its ID and connection and then nothing happens.
Every time to restart the cluster I have to shutdown datanodes and restart with --initial reimporting all data.
This is the procedure I follow:
ndb_restore --ndb_connectstring=192.168.1.1,192.168.1.2 --nodeid=100 --disable-indexes --restore_meta --backupid=5 --backup_path="BACKUP PATH" --verbose=255
(Only one time to import schema)
ndb_restore --ndb_connectstring=192.168.1.1,192.168.1.2 --nodeid=100 --disable-indexes --restore_data --backupid=5 --backup_path="BACKUP_PATH" --verbose=255
(One for each dataNode 100,101,102,103)
Schema restores succesfully but sometimes data restore freezes. It's strange becuase it freezes at different times and points.
This is the configuration:
SharedGlobalMemory=1G
MaxNoOfExecutionThreads=4
MaxNoOfConcurrentOperations=1000000
MaxNoOfTables=4096
TransactionDeadlockDetectionTimeout=100000000
TransactionInactiveTimeout=100000000
MaxNoOfTriggers=4096 # Default is 768
FragmentLogFileSize=512M
NoOfFragmentLogFiles=100
RealTimeScheduler=1
MinDiskWriteSpeed=10M
MaxDiskWriteSpeed=50M
[ndb_mgmd]
HostName=192.168.1.1
NodeId=1
DataDir=/var/lib/mysql-cluster
[ndb_mgmd]
HostName=192.168.1.2
NodeId=2
DataDir=/var/lib/mysql-cluster
[ndbd]
HostName=192.168.1.200
NodeId=100
DataDir=/usr/local/mysql/data
[ndbd]
HostName=192.168.1.201
NodeId=101
DataDir=/usr/local/mysql/data
[ndbd]
HostName=192.168.1.202
NodeId=102
DataDir=/usr/local/mysql/data
[ndbd]
HostName=192.168.1.203
NodeId=103
DataDir=/usr/local/mysql/data
[mysqld]
NodeId=50
HostName=192.168.1.100
[mysqld]
NodeId=51
HostName=192.168.1.101
Process stacks at points like this:
Processing data in table: DBNAME/def/TABLENAME(211) fragment 0
When I start a restore and it freezes this is the log result in the dataNode
DataNode log:
2021-02-17 12:24:27 [ndbd] INFO -- Node 50 disconnected in state: 0
2021-02-17 12:25:32 [ndbd] INFO -- Node 50 disconnected in recv with errnum: 104 in state: 0
2021-02-17 12:26:20 [ndbd] INFO -- Node 50 disconnected in state: 0
2021-02-17 12:27:23 [ndbd] INFO -- Node 50 disconnected in state: 0
2021-02-17 12:28:50 [ndbd] INFO -- Node 50 disconnected in state: 0
2021-02-17 12:29:25 [ndbd] INFO -- Node 50 disconnected in state: 0
2021-02-17 12:30:08 [ndbd] INFO -- Node 50 disconnected in recv with errnum: 104 in state: 0
2021-02-17 12:30:34 [ndbd] INFO -- part: 1 : time to complete: 1
2021-02-17 12:30:39 [ndbd] INFO -- Node 50 disconnected in state: 0
2021-02-17 12:33:00 [ndbd] INFO -- Node 50 disconnected in state: 0
Any help would be greatly appreciated, thanks a lot!
Giuseppe Rossi