I need to load bunch of data from csv like files to MYSQL NDB tables.
Q1. Uploading data in MyISAM/InnoDb takes less than a minute for 2M rows. For NDB, it takes 700 seconds. How can I make this perform for NDB? The setup is a single node, with no replication (pseudo cluster)
Q2. Uploading data to MYISAM and then ALERT TABLE to NDB works like magic for 10M rows, but goes for a toss when it is 100M records. ( I just abort the script after 3 hours)
Q3. I got ‘table full’ error even when the table was using disk storage. In the case of disk storage, why do I get ‘table full error?
Q4. I want to upload 180M rows, But I am not able to cross 60M rows ( table full error).
Configuration:
[ndbd default]
noofreplicas=1
MaxBufferedEpochs=10000
TimeBetweenEpochsTimeout=256000
TransactionDeadlockDetectionTimeout=3600000
datadir=C:\MYSQLcluster\Data
[ndbd]
DataMemory=16G
IndexMemory=4G
MaxNoOfConcurrentOperations=10000
MaxNoOfLocalOperations=10000
hostname=localhost
NodeId=2
[ndb_mgmd]
NodeId=1
hostname=localhost
[mysqld]
NodeId=101
hostname=localhost
Hardware:
32GB RAM VM
Script
use db1;
DROP TABLE IF EXISTS T1;
CREATE TABLE tb1(
C1 char(36) NOT NULL,
C2 char(36) NOT NULL,
CDate1 timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
CDate2 timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
icv tinyint NOT NULL default 0,
C6 char(20) NOT NULL)tablespace ts_1 storage disk engine =ndb ;
LOAD DATA LOCAL INFILE 'c:\\data\\bcpout\\File2013-03-12.c'
IGNORE
INTO TABLE T1
FIELDS TERMINATED BY '*FT*'
LINES TERMINATED BY '*LT*'
(C1,C2,CDate1,Cdate2,icv,C6);
Q1. Uploading data in MyISAM/InnoDb takes less than a minute for 2M rows. For NDB, it takes 700 seconds. How can I make this perform for NDB? The setup is a single node, with no replication (pseudo cluster)
Q2. Uploading data to MYISAM and then ALERT TABLE to NDB works like magic for 10M rows, but goes for a toss when it is 100M records. ( I just abort the script after 3 hours)
Q3. I got ‘table full’ error even when the table was using disk storage. In the case of disk storage, why do I get ‘table full error?
Q4. I want to upload 180M rows, But I am not able to cross 60M rows ( table full error).
Configuration:
[ndbd default]
noofreplicas=1
MaxBufferedEpochs=10000
TimeBetweenEpochsTimeout=256000
TransactionDeadlockDetectionTimeout=3600000
datadir=C:\MYSQLcluster\Data
[ndbd]
DataMemory=16G
IndexMemory=4G
MaxNoOfConcurrentOperations=10000
MaxNoOfLocalOperations=10000
hostname=localhost
NodeId=2
[ndb_mgmd]
NodeId=1
hostname=localhost
[mysqld]
NodeId=101
hostname=localhost
Hardware:
32GB RAM VM
Script
use db1;
DROP TABLE IF EXISTS T1;
CREATE TABLE tb1(
C1 char(36) NOT NULL,
C2 char(36) NOT NULL,
CDate1 timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
CDate2 timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
icv tinyint NOT NULL default 0,
C6 char(20) NOT NULL)tablespace ts_1 storage disk engine =ndb ;
LOAD DATA LOCAL INFILE 'c:\\data\\bcpout\\File2013-03-12.c'
IGNORE
INTO TABLE T1
FIELDS TERMINATED BY '*FT*'
LINES TERMINATED BY '*LT*'
(C1,C2,CDate1,Cdate2,icv,C6);