Discalimer: this is a test cluster, even if inserted in a production environment, so please do not answer with best practices: I know them, it's just a test.

So, I'm running a 3 datanodes cluster on 3 machines (ndb_mgmd in the one of these machines), and moreover all those 3 machines are in the same chassis (it's not a blade, it's a strange thing from Dell), so they share the same power supplies. Now, for some unknown reason, the whole chassis restarted (we guess some kind of power failure), so the 3 machines were rebooted at the exact same time, and now I'm with an unstartable cluster cause ndbmtd keeps crashing on every node. This is the error utput from one

Time: Monday 10 December 2012 - 12:41:38
Status: Ndbd file system error, restart node initial
Message: Invalid LCP (Ndbd file system inconsistency error, please report a bug)
Error: 2352
Error data: T21F9
Error object: RESTORE (Line: 1283) 0x00000002
Program: ndbmtd
Pid: 14239 thr: 3
Version: mysql-5.5.27 ndb-7.2.8
Trace: /usr/local/mysql/data/ndb_2_trace.log.10 [t1..t5]
***EOM***

The filesystem seems ok in every node... so is this a bug or a normal behaviour in a failing cluster?

EDIT: I have the same error even if starting with ndbmtd --initial which should delete all the local data (LCP etc) on disk.

(I think, this is a wrong place for my topic, I posted here by mistake.)

Hello,


I tested a replication scema, on Amazon EC2, master is a m1.Xlarge instance and slave (right now) a m1.large one. On Master, queryes are predominantly WRITES, ads occasionaly READS. I monitored Master, and it is receive at least 120 Connections/sec , and 960 Queryes/sec. Master Server is not very load with this traffic. Slave it is insignificant load!
When I look into processlist on master, I found the replication process always as "State: Writing to net" at "Command: Binlog Dump". The slave is always in "Queueing master event to the relay log" state. As a result, Slave is always behind master!

We use Provisioned IOPS EBS (200 and 400 IOPS), different EBS for data and binlog / relaylog. IO on EBS is very low.

Some buffers setings are:
net_buffer_length = 1M
read_buffer_size = 2M
sort_buffer_size = 8M

I do not understand why the replication process is very very slow, and nor master and slave is not loaded. It is possible to speed up replication process on slave?
Thank You

Hello,

We are using MySQL Cluster 7.1.18.
Application is using Native C API to select from a table on ndbcluster engine. There is no other update operation on the same.

We are getting following error on the select operation very frequently:

Error Code: 1205
Error Message: Lock wait timeout exceeded; try restarting transaction

Our Config files are :

=================================
my.cnf
=================================

[mysqld]
server-id=12
ndbcluster
ndb-connectstring="x.x.x.167:1186"
port=3306
ndb_cluster_connection_pool=8
ndb-wait-setup=45
socket=/var/lib/mysql/mysql.sock
skip-name-resolve
max-connections=400
long_query_time=1
ndb_cache_check_time=500
ndb_extra_logging=0
ndb_index_stat_enable=ON
#ndb_index_stat_update_freq=100
ndb_log_bin=OFF
interactive_timeout=604800
wait_timeout=604800
[mysql_cluster]
ndb-connectstring="x.x.x.167:1186"

=================================
config.ini
=================================
[TCP DEFAULT]
SendBufferMemory=2M
ReceiveBufferMemory=2M

[NDB_MGMD DEFAULT]
PortNumber=1186
Datadir=/rancore2/mysql-cluster

[NDB_MGMD]
NodeId=1
Hostname=x.x.x.167
LogDestination=FILE:filename=ndb_1_cluster.log,maxsize=10000000,maxfiles=6
ArbitrationRank=1

[NDBD DEFAULT]
NoOfReplicas=1
Datadir=/rancore/mysql-cluster/data1
#FileSystemPathDD=/rancore/mysql-cluster
DataMemory=512M
IndexMemory=64M
LockPagesInMainMemory=1

MaxNoOfConcurrentOperations=32768
MaxNoOfConcurrentTransactions=16384

StringMemory=25
MaxNoOfTables=4096
MaxNoOfOrderedIndexes=1024
MaxNoOfUniqueHashIndexes=1024
MaxNoOfAttributes=10000
MaxNoOfTriggers=14336
DiskCheckpointSpeedInRestart=50M
FragmentLogFileSize=128M
InitFragmentLogFiles=SPARSE
NoOfFragmentLogFiles=3
RedoBuffer=4M

TimeBetweenLocalCheckpoints=20
TimeBetweenEpochsTimeout=32000
TimeBetweenGlobalCheckpoints=2000
TimeBetweenEpochs=100


### Heartbeating
HeartbeatIntervalDbDb=15000
HeartbeatIntervalDbApi=15000

### Params for setting logging
MemReportFrequency=30
BackupReportFrequency=10
LogLevelStartup=15
LogLevelShutdown=15
LogLevelCheckpoint=8
LogLevelNodeRestart=15

### Params for BACKUP
BackupMaxWriteSize=1M
BackupDataBufferSize=16M
BackupLogBufferSize=4M
BackupMemory=20M

### Params for ODIRECT
#Reports indicates that odirect=1 can cause io errors (os err code 5) on some systems. You must test.
#ODirect=1

### Watchdog
TimeBetweenWatchdogCheckInitial=60000

### TransactionInactiveTimeout - should be enabled in Production
TransactionInactiveTimeout=60000
### New 7.1.10 redo logging parameters
RedoOverCommitCounter=3
RedoOverCommitLimit=20
### REALTIME EXTENSIONS
#RealTimeScheduler=1
### REALTIME EXTENSIONS FOR 6.3 ONLY
#SchedulerExecutionTimer=80
#SchedulerSpinTimer=40

### DISK DATA
SharedGlobalMemory=20M
DiskPageBufferMemory=20M
BatchSizePerLocalScan=512
[NDBD]
NodeId=2
Hostname=x.x.x.167

### REALTIME ASPECTS - THREAD BINDING
#LockExecuteThreadToCPU=X
#LockMaintThreadsToCPU=Y

[MYSQLD DEFAULT]
DefaultOperationRedoProblemAction=QUEUE
BatchSize=512
#BatchByteSize=2048K
#MaxScanBatchSize=2048K

[MYSQLD]
[MYSQLD]
[MYSQLD]
[MYSQLD]
[MYSQLD]
[MYSQLD]

Please Reply for the same ASAP.
Thanks.

Hi MySQL,

Version 7.2.8

This is a dedicated mysqld node serving ndb. This handles an average of 120 queries per second.

Three times in the last two weeks I have experienced a lock. In this state, any query regarding an NDB table will not return. This can only be cleared using 'kill -9' on the mysqld process.

This summarised process list shows the initial query suspected of causing the lock: (Full process list further on.)

+-------+---------+------+-----------------------------------+------
| Id | Command | Time | State | Info
+-------+---------+------+-----------------------------------+------
| 1 | Daemon | 0 | Waiting for event from ndbcluster | NULL
| 21455 | Query | 657 | Waiting for table level lock | INSERT IGNORE INTO fl_state ...
| 21505 | Query | 0 | NULL | show full processlist
+-------+---------+------+-----------------------------------+------

After which queries will queue indefinitely:

+-------+---------+------+-----------------------------------+------
| Id | Command | Time | State | Info
+-------+---------+------+-----------------------------------+------
| 1 | Daemon | 0 | Waiting for event from ndbcluster | NULL
| 21455 | Query | 1820 | Waiting for table level lock | INSERT IGNORE INTO fl_state ...
| 21483 | Query | 620 | System lock | UPDATE fl_state SET ...
| 21567 | Query | 620 | closing tables | UPDATE fl_state SET ...
| 21578 | Query | 501 | System lock | INSERT IGNORE INTO fl_state ...
| 21589 | Query | 380 | System lock | INSERT IGNORE INTO fl_state ...
| 21600 | Query | 260 | System lock | INSERT IGNORE INTO fl_state ...
| 21611 | Query | 140 | System lock | INSERT IGNORE INTO fl_state ...
| 21622 | Query | 21 | System lock | INSERT IGNORE INTO fl_state ...
| 21625 | Query | 0 | NULL | show processlist
+-------+---------+------+-----------------------------------+-------

The logs on the 'ndb_mgmd' and 'ndb' nodes do not show any activity at these times.

This seems to be some bug in 'mysqld'. However I would be interested in knowing anything we could do to help avoid this problem, as it's rather annoying having to 'kill -9' a production database every few days.

Thanks to anybody who can help.

Ben


Full process lists:



+-------+-------------+---------------------------------+------+---------+------+-----------------------------------+----------------------------------------------------------------------------+
| Id | User | Host | db | Command | Time | State | Info |
+-------+-------------+---------------------------------+------+---------+------+-----------------------------------+----------------------------------------------------------------------------+
| 1 | system user | | | Daemon | 0 | Waiting for event from ndbcluster | NULL |
| 34 | webservice | hp-tm-01.roadtech.private:50436 | ndb | Sleep | 18 | | NULL |
| 37 | webservice | hp-tm-01.roadtech.private:50443 | ndb | Sleep | 5 | | NULL |
| 38 | webservice | hp-tm-01.roadtech.private:50445 | ndb | Sleep | 3 | | NULL |
| 11909 | webservice | hp-tm-37:59063 | ndb | Sleep | 4 | | NULL |
| 11910 | webservice | 172.16.14.64:50801 | ndb | Sleep | 2 | | NULL |
| 15820 | webservice | hp-tm-50.roadtech.private:59351 | ndb | Sleep | 17 | | NULL |
| 15827 | webservice | hp-tm-54.roadtech.private:39689 | ndb | Sleep | 57 | | NULL |
| 15833 | webservice | hp-tm-58.roadtech.private:49258 | ndb | Sleep | 42 | | NULL |
| 15846 | webservice | hp-tm-62.roadtech.private:53575 | ndb | Sleep | 50 | | NULL |
| 15866 | webservice | hp-tm-25.roadtech.private:57639 | ndb | Sleep | 10 | | NULL |
| 15881 | webservice | hp-tm-30.roadtech.private:59779 | ndb | Sleep | 6 | | NULL |
| 15894 | webservice | hp-tm-51.roadtech.private:40458 | ndb | Sleep | 52 | | NULL |
| 15901 | webservice | hp-tm-55.roadtech.private:59112 | ndb | Sleep | 44 | | NULL |
| 15924 | webservice | hp-tm-63.roadtech.private:43354 | ndb | Sleep | 44 | | NULL |
| 15930 | webservice | hp-tm-26.roadtech.private:46992 | ndb | Sleep | 41 | | NULL |
| 15988 | webservice | hp-tm-31.roadtech.private:59777 | ndb | Sleep | 56 | | NULL |
| 18944 | webservice | hp-tm-75.roadtech.private:36674 | ndb | Sleep | 11 | | NULL |
| 19359 | webservice | hp-tm-27.roadtech.private:38875 | ndb | Sleep | 45 | | NULL |
| 19608 | webservice | hp-tm-72.roadtech.private:39805 | ndb | Sleep | 19 | | NULL |
| 20026 | webservice | hp-tm-36.roadtech.private:60018 | ndb | Sleep | 50 | | NULL |
| 20359 | webservice | hp-tm-59.roadtech.private:53276 | ndb | Sleep | 15 | | NULL |
| 20442 | webservice | hp-tm-73.roadtech.private:42660 | ndb | Sleep | 5 | | NULL |
| 20443 | webservice | hp-tm-71.roadtech.private:52507 | ndb | Sleep | 26 | | NULL |
| 20444 | webservice | hp-tm-70.roadtech.private:50333 | ndb | Sleep | 61 | | NULL |
| 20613 | webservice | hp-tm-91.roadtech.private:34992 | ndb | Sleep | 41 | | NULL |
| 20615 | webservice | hp-tm-90.roadtech.private:55923 | ndb | Sleep | 13 | | NULL |
| 20862 | webservice | hp-tm-74.roadtech.private:58417 | ndb | Sleep | 55 | | NULL |
| 20865 | webservice | hp-tm-35.roadtech.private:37240 | ndb | Sleep | 5 | | NULL |
| 21034 | webservice | hp-tm-28.roadtech.private:56938 | ndb | Sleep | 25 | | NULL |
| 21455 | webservice | hp-tm-01.roadtech.private:37450 | ndb | Query | 657 | Waiting for table level lock | INSERT IGNORE INTO fl_state (terminal_id) VALUES ( 'DYNA_WHMA20_PX58EEV' ) |
| 21483 | webservice | hp-tm-01.roadtech.private:57172 | ndb | Sleep | 58 | | NULL |
| 21505 | root | localhost | NULL | Query | 0 | NULL | show full processlist |
+-------+-------------+---------------------------------+------+---------+------+-----------------------------------+----------------------------------------------------------------------------+

show processlist;
+-------+-------------+---------------------------------+------+---------+------+-----------------------------------+------------------------------------------------------------------------------------------------------+
| Id | User | Host | db | Command | Time | State | Info |
+-------+-------------+---------------------------------+------+---------+------+-----------------------------------+------------------------------------------------------------------------------------------------------+
| 1 | system user | | | Daemon | 0 | Waiting for event from ndbcluster | NULL |
| 34 | webservice | hp-tm-01.roadtech.private:50436 | ndb | Sleep | 40 | | NULL |
| 37 | webservice | hp-tm-01.roadtech.private:50443 | ndb | Sleep | 28 | | NULL |
| 38 | webservice | hp-tm-01.roadtech.private:50445 | ndb | Sleep | 21 | | NULL |
| 11909 | webservice | hp-tm-37:59063 | ndb | Sleep | 27 | | NULL |
| 11910 | webservice | 172.16.14.64:50801 | ndb | Sleep | 25 | | NULL |
| 15820 | webservice | hp-tm-50.roadtech.private:59351 | ndb | Sleep | 29 | | NULL |
| 15827 | webservice | hp-tm-54.roadtech.private:39689 | ndb | Sleep | 11 | | NULL |
| 15833 | webservice | hp-tm-58.roadtech.private:49258 | ndb | Sleep | 3 | | NULL |
| 15846 | webservice | hp-tm-62.roadtech.private:53575 | ndb | Sleep | 27 | | NULL |
| 15866 | webservice | hp-tm-25.roadtech.private:57639 | ndb | Sleep | 5 | | NULL |
| 15881 | webservice | hp-tm-30.roadtech.private:59779 | ndb | Sleep | 14 | | NULL |
| 15894 | webservice | hp-tm-51.roadtech.private:40458 | ndb | Sleep | 10 | | NULL |
| 15901 | webservice | hp-tm-55.roadtech.private:59112 | ndb | Sleep | 54 | | NULL |
| 15924 | webservice | hp-tm-63.roadtech.private:43354 | ndb | Sleep | 7 | | NULL |
| 15930 | webservice | hp-tm-26.roadtech.private:46992 | ndb | Sleep | 18 | | NULL |
| 15988 | webservice | hp-tm-31.roadtech.private:59777 | ndb | Sleep | 42 | | NULL |
| 18944 | webservice | hp-tm-75.roadtech.private:36674 | ndb | Sleep | 30 | | NULL |
| 19359 | webservice | hp-tm-27.roadtech.private:38875 | ndb | Sleep | 60 | | NULL |
| 19608 | webservice | hp-tm-72.roadtech.private:39805 | ndb | Sleep | 12 | | NULL |
| 20026 | webservice | hp-tm-36.roadtech.private:60018 | ndb | Sleep | 13 | | NULL |
| 20359 | webservice | hp-tm-59.roadtech.private:53276 | ndb | Sleep | 23 | | NULL |
| 20442 | webservice | hp-tm-73.roadtech.private:42660 | ndb | Sleep | 20 | | NULL |
| 20443 | webservice | hp-tm-71.roadtech.private:52507 | ndb | Sleep | 30 | | NULL |
| 20444 | webservice | hp-tm-70.roadtech.private:50333 | ndb | Sleep | 5 | | NULL |
| 20613 | webservice | hp-tm-91.roadtech.private:34992 | ndb | Sleep | 52 | | NULL |
| 20615 | webservice | hp-tm-90.roadtech.private:55923 | ndb | Sleep | 26 | | NULL |
| 20862 | webservice | hp-tm-74.roadtech.private:58417 | ndb | Sleep | 14 | | NULL |
| 20865 | webservice | hp-tm-35.roadtech.private:37240 | ndb | Sleep | 9 | | NULL |
| 21034 | webservice | hp-tm-28.roadtech.private:56938 | ndb | Sleep | 21 | | NULL |
| 21455 | webservice | hp-tm-01.roadtech.private:37450 | ndb | Query | 1820 | Waiting for table level lock | INSERT IGNORE INTO fl_state (terminal_id) VALUES ( 'DYNA_WHMA20_PX58EEV' ) |
| 21483 | webservice | hp-tm-01.roadtech.private:57172 | ndb | Query | 620 | System lock | UPDATE fl_state SET tally = tally + 1, md_t = '2012-12-11 17:18:48' , md_valid_fix = TRUE, md_serve |
| 21505 | root | localhost | NULL | Sleep | 6 | | NULL |
| 21567 | webservice | hp-tm-01.roadtech.private:38467 | ndb | Query | 620 | closing tables | UPDATE fl_state SET tally = tally + 1, md_t = '2012-12-11 17:21:14' , md_valid_fix = TRUE, md_serve |
| 21578 | webservice | hp-tm-01.roadtech.private:38541 | ndb | Query | 501 | System lock | INSERT IGNORE INTO fl_state (terminal_id) VALUES ( 'DYNA_WHMA20_KP60EBX' ) |
| 21589 | webservice | hp-tm-01.roadtech.private:54332 | ndb | Query | 380 | System lock | INSERT IGNORE INTO fl_state (terminal_id) VALUES ( 'DYNA_WHMA20_SF58BHE' ) |
| 21600 | webservice | hp-tm-01.roadtech.private:54404 | ndb | Query | 260 | System lock | INSERT IGNORE INTO fl_state (terminal_id) VALUES ( 'DYNA_WHMA20_KP60EBX' ) |
| 21611 | webservice | hp-tm-01.roadtech.private:54560 | ndb | Query | 140 | System lock | INSERT IGNORE INTO fl_state (terminal_id) VALUES ( 'DYNA_WHMA20_SF12 CXE' ) |
| 21622 | webservice | hp-tm-01.roadtech.private:54634 | ndb | Query | 21 | System lock | INSERT IGNORE INTO fl_state (terminal_id) VALUES ( 'DYNA_WHMA20_KP60EBX' ) |
| 21625 | root | localhost | NULL | Query | 0 | NULL | show processlist |
+-------+-------------+---------------------------------+------+---------+------+-----------------------------------+------------------------------------------------------------------------------------------------------+

Trying to upgrade an existing environment from MySQL Cluster 7.2.7 to 7.2.9. The cluster will start and all the nodes are active but fails and shutsdown when trying to access tables.

From /data/mysqlcluster/ndb_1_cluster.log I see these errors.
2012-12-12 11:20:18 [MgmtSrvr] ALERT -- Node 10: Forced node shutdown completed. Caused by error 2301: 'Assertion(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'.
2012-12-12 11:20:28 [MgmtSrvr] ALERT -- Node 11: Forced node shutdown completed. Caused by error 2301: 'Assertion(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'.

From /data/mysql/error.log I see several of these error messages.
121212 11:21:46 [Warning] NDB: Could not acquire global schema lock (4009)Cluster Failure

Also
121212 11:51:07 [ERROR] /usr/local//mysql/bin//mysqld: Incorrect information in file: './availability/availabilitydataloaderstate.frm'

Hi, there.

How can I delete the table in Memory after using ALTER TABLE and move the table to disk data?
It seems after I use ALTER TABLE to move the exist table in DataMemory to disk, using cmd:
ALTER TABLE tb01 TABLESPACE ts01 STORAGE DISK NDB;

The DamaMemory increased, but no space is used in the tablespace.
And from the information_schema.partision, we can see the table exists in the tablespace.

Why is like this?

So if we want to move a table from Memory to Disk, do we need to DROP the table first and use "CREATE TABLE ...TABLESPACE ts01 STORAGE DISK NDB" to store it on the disk?

Thanks very much

What do you think about using ssd disks with mysql cluster? I read microsoft study, that for SQL server it is not very reliable (SSD's lifetime is significantly , it is the same with ndbd?

Any help would be much appreciated!

We are evaluating mysql cluster for a large/high profile project. We set it up yesterday, and all was going well. This morning we find this though:

The table claims not to exist here:

mysql (root@localhost)> select * from items limit 10;
ERROR 1032 (HY000): Can't find record in 'items'
mysql (root@localhost)> show errors;
+-------+------+------------------------------+
| Level | Code | Message |
+-------+------+------------------------------+
| Error | 1032 | Can't find record in 'items' |
+-------+------+------------------------------+
1 row in set (0.00 sec)



But the table exists here:

mysql (root@localhost)> show tables;
+---------------+
| Tables_in_mbe |
+---------------+
| aspects |
| item_aspects |
| items |
| users |
+---------------+
4 rows in set (0.03 sec)


Table exists here:

mysql (root@localhost)> show create table items;
---------------------------------------+
| Table | Create Table |
| items | CREATE TABLE `items` (
`id` bigint(20) NOT NULL,

BLA BLA BLA
) ENGINE=ndbcluster DEFAULT CHARSET=utf8 |
---------------+
1 row in set (0.00 sec)


The table exists here:

mysql (root@localhost)> select count(*) from items;
+----------+
| count(*) |
+----------+
| 5507351 |
+----------+
1 row in set (0.00 sec)


1) how can this happen?!?!
2) how can I fix it?

ndb_mgm> show
Connected to Management Server at: localhost:1186
Cluster Configuration
---------------------
[ndbd(NDB)] 6 node(s)
id=2 @10.86.134.44 (mysql-5.5.28 ndb-7.2.9, Nodegroup: 0, Master)
id=3 @10.86.130.83 (mysql-5.5.28 ndb-7.2.9, Nodegroup: 0)
id=4 @10.86.140.93 (mysql-5.5.28 ndb-7.2.9, Nodegroup: 1)
id=5 @10.86.138.86 (mysql-5.5.28 ndb-7.2.9, Nodegroup: 1)
id=6 @10.86.138.91 (mysql-5.5.28 ndb-7.2.9, Nodegroup: 2)
id=7 @10.86.140.89 (mysql-5.5.28 ndb-7.2.9, Nodegroup: 2)

[ndb_mgmd(MGM)] 1 node(s)
id=1 @10.86.138.85 (mysql-5.5.28 ndb-7.2.9)

[mysqld(API)] 1 node(s)
id=8 @10.86.138.85 (mysql-5.5.28 ndb-7.2.9)

ndb_mgm> all report memory
Node 2: Data usage is 5%(94170 32K pages of total 1638400)
Node 2: Index usage is 3%(21814 8K pages of total 655392)
Node 3: Data usage is 5%(94168 32K pages of total 1638400)
Node 3: Index usage is 3%(21814 8K pages of total 655392)
Node 4: Data usage is 5%(94235 32K pages of total 1638400)
Node 4: Index usage is 3%(21809 8K pages of total 655392)
Node 5: Data usage is 5%(94235 32K pages of total 1638400)
Node 5: Index usage is 3%(21809 8K pages of total 655392)
Node 6: Data usage is 5%(94091 32K pages of total 1638400)
Node 6: Index usage is 3%(21798 8K pages of total 655392)
Node 7: Data usage is 5%(94088 32K pages of total 1638400)
Node 7: Index usage is 3%(21800 8K pages of total 655392)



-----------------------ndb_mgm config------------------------------
[mysqld]
ndbcluster
ndb-connectstring=mc-master-1-phx

[mysql_cluster]
ndb-connectstring=mc-master-1-phx

09:53:51 aspitzer@phx7b02c-718516|mc-master-1-phx:~ > more /data/mysql-cluster/conf/mgmt.phx
[tcp default]
SendBufferMemory=2M
ReceiveBufferMemory=2M

[ndb_mgmd]
hostname=mc-master-1-phx
datadir=/data/mysql-cluster-data/mgmt

[ndbd default]
datadir=/data/mysql-cluster-data/ndb
NoOfReplicas=2
#LockPagesInMainMemory=1
DataMemory=50G
IndexMemory=5G
#ODirect=1
MaxNoOfConcurrentOperations=100000

SchedulerSpinTimer=400
SchedulerExecutionTimer=100
RealTimeScheduler=1
TimeBetweenGlobalCheckpoints=1000
TimeBetweenEpochs=200
DiskCheckpointSpeed=10M
DiskCheckpointSpeedInRestart=100M
RedoBuffer=32M
CompressedLCP=1
CompressedBackup=1
#this all came from here:
# http://dev.mysql.com/doc/refman/5.5/en/mysql-cluster-config-starting.html

[ndbd]
hostname=mc-1-phx
datadir=/data/mysql-cluster-data/ndb
LockExecuteThreadToCPU=1
LockMaintThreadsToCPU=0

[ndbd]
hostname=mc-2-phx
datadir=/data/mysql-cluster-data/ndb
LockExecuteThreadToCPU=1
LockMaintThreadsToCPU=0

[ndbd]
hostname=mc-3-phx
datadir=/data/mysql-cluster-data/ndb
LockExecuteThreadToCPU=1
LockMaintThreadsToCPU=0

[ndbd]
hostname=mc-4-phx
datadir=/data/mysql-cluster-data/ndb
LockExecuteThreadToCPU=1
LockMaintThreadsToCPU=0

[ndbd]
hostname=mc-5-phx
datadir=/data/mysql-cluster-data/ndb
LockExecuteThreadToCPU=1
LockMaintThreadsToCPU=0

[ndbd]
hostname=mc-6-phx
datadir=/data/mysql-cluster-data/ndb
LockExecuteThreadToCPU=1
LockMaintThreadsToCPU=0

[mysqld]
hostname=mc-master-1-phx
----------------------------------------------------


-----------------ndbd config---------------
[mysqld]
ndbcluster
ndb-connectstring=mc-master-1-phx

[mysql_cluster]
ndb-connectstring=mc-master-1-phx
---------------------------------------


------------ MYSQL config -----------------
[mysqld]
port = 3306
socket = /tmp/mysql.sock
ndbcluster
ndb-connectstring=mc-master-1-phx

max_connect_errors = 1000
max_connections = 5000
innodb_buffer_pool_size = 16G
key_buffer_size = 16G
table_open_cache = 512
sort_buffer_size = 4M
read_buffer_size = 4M
read_rnd_buffer_size = 8M
myisam_sort_buffer_size = 128M
thread_cache_size = 8
query_cache_size = 32M
# Try number of CPU's*2 for thread_concurrency
thread_concurrency = 8

max_allowed_packet = 16M
default-storage-engine = ndbcluster


[mysql_cluster]
ndb-connectstring=mc-master-1-phx
---------------------------------------------

Hello,

My Cluster meet a problem:
NDB: error (1) Configuration error: Error: Could not alloc node id at 192.168.201.171 port 1186: Connection with id 74 done from wrong host ip 192.168.201.33, expected 192.168.201.34(192.168.201.34).

Who can help me to solve~
My Cluster structure as below:

A total of four machines - Every service starts on a different network card
1. server node1 & server node2 & management node
server node1 ip: 192.168.201.31
server node2 ip: 192.168.201.32
management node ip: 192.168.201.171

2. server node3 & server node4
server node3 ip: 192.168.201.33
server node4 ip: 192.168.201.34

3. data node1
data node1 ip: 192.168.201.51

4. data node2
data node2 ip: 192.168.201.52

The managament node config.ini as below:
[ndbd default]
# Options affecting ndbd processes on all data nodes:
NoOfReplicas=2 # Number of replicas
DataMemory=200G # How much memory to allocate for data storage
IndexMemory=20G # How much memory to allocate for index storage
# For DataMemory and IndexMemory, we have used the
# default values. Since the "world" database takes up
# only about 500KB, this should be more than enough for
# this example Cluster setup.

[MYSQLD DEFAULT]
[NDB_MGMD DEFAULT]
[tcp default]
# TCP/IP options:
#portnumber=2202 # This the default; however, you can use any
# port that is free for all the hosts in the cluster
# Note: It is recommended that you do not specify the port
# number at all and simply allow the default value to be used
# instead

[ndb_mgmd]
# Management process options:
NodeId = 21
hostname=192.168.201.171 # Hostname or IP address of MGM node
datadir=/var/lib/mysql-cluster # Directory for MGM node log files

[ndbd]
# Options for data node "A":
# (one [ndbd] section per data node)
NodeId = 1
hostname=192.168.201.51 # Hostname or IP address
datadir=/usr/local/mysql/data # Directory for this data node's data files

[ndbd]
NodeId = 2
# Options for data node "B":
hostname=192.168.201.52 # Hostname or IP address
datadir=/usr/local/mysql/data # Directory for this data node's data files

[mysqld]
# SQL node options:
NodeId = 71
hostname=192.168.201.31 # Hostname or IP address
# (additional mysqld connections can be
# specified for this node for various
# purposes such as running ndb_restore)

[mysqld]
# SQL node options:
NodeId = 72
hostname=192.168.201.32 # Hostname or IP address
# (additional mysqld connections can be
# specified for this node for various
# purposes such as running ndb_restore)

[mysqld]
# SQL node options:
NodeId = 73
hostname=192.168.201.33 # Hostname or IP address
# (additional mysqld connections can be
# specified for this node for various
# purposes such as running ndb_restore)

[mysqld]
# SQL node options:
NodeId = 74
hostname=192.168.201.34 # Hostname or IP address
# (additional mysqld connections can be
# specified for this node for various
# purposes such as running ndb_restore)


The server node my.cnf set as below:
server node1:
[mysqld]
# Options for mysqld process:
ndbcluster # run NDB storage engine
ndb-nodeid=71
bind-address=192.168.201.31
server-id=71
pid-file=/usr/local/mysql-cluster-gpl-7.2.8-net31/mysql31.pid

server node2:
[mysqld]
# Options for mysqld process:
ndbcluster # run NDB storage engine
ndb-nodeid=72
bind-address=192.168.201.32
server-id=72
pid-file=/usr/local/mysql-cluster-gpl-7.2.8-net32/mysql32.pid

server node3:
[mysqld]
# Options for mysqld process:
ndbcluster # run NDB storage engine
ndb-nodeid=73
bind-address=192.168.201.33
server-id=73
pid-file=/usr/local/mysql-cluster-gpl-7.2.8-net33/mysql33.pid

ndb-connectstring=192.168.201.171 # location of management server

# Options for ndbd process:
[mysql_cluster]
ndb-connectstring=192.168.201.171 # location of management server

server node4:
[mysqld]
# Options for mysqld process:
ndbcluster # run NDB storage engine
ndb-nodeid=74
bind-address=192.168.201.34
server-id=74
pid-file=/usr/local/mysql-cluster-gpl-7.2.8-net34/mysql34.pid

ndb-connectstring=192.168.201.171 # location of management server

# Options for ndbd process:
[mysql_cluster]
ndb-connectstring=192.168.201.171 # location of management server

The management can't catch the server node 4 - 192.168.201.34:

[root@mysqlc-s0102-m01 bin]# ./ndb_mgm -e show
Connected to Management Server at: localhost:1186
Cluster Configuration
---------------------
[ndbd(NDB)] 2 node(s)
id=1 @192.168.201.51 (mysql-5.5.27 ndb-7.2.8, Nodegroup: 0, Master)
id=2 @192.168.201.52 (mysql-5.5.27 ndb-7.2.8, Nodegroup: 0)

[ndb_mgmd(MGM)] 1 node(s)
id=21 @192.168.201.171 (mysql-5.5.27 ndb-7.2.8)

[mysqld(API)] 4 node(s)
id=71 @192.168.201.31 (mysql-5.5.27 ndb-7.2.8)
id=72 @192.168.201.32 (mysql-5.5.27 ndb-7.2.8)
id=73 @192.168.201.33 (mysql-5.5.27 ndb-7.2.8)
id=74 (not connected, accepting connect from 192.168.201.34)

check log is
NDB: error (1) Configuration error: Error: Could not alloc node id at 192.168.201.171 port 1186: Connection with id 74 done from wrong host ip 192.168.201.33, expected 192.168.201.34(192.168.201.34).

Please help me~~~~

Thanks very much!!

Getting the exception:
java.sql.SQLException: Can't find record in 'linkedavailableroute'

This is occuring when loading records from a csv file using
load data local infile '{fileName}' replace into table {tableName} fields terminated by ',';

Environment is mysql-5.5.28 ndb-7.2.9

MySql,

API mysql-5.5.27 ndb-7.2.8

I experienced an outage on a 'mysqld' process last night, complaining on an error 257.

This caused loss of our service for half an hour until it sself-recovered. But as such I need to know what happened and how to avoid in the future. I have attached the logs.

If any user can explain these, I would be greatful.

Regards,

Ben Clewett.


mysqld:

121219 18:19:46 [ERROR] Got error 157 when reading table './ndb/fl_state'
121219 18:19:46 [Note] NDB Binlog: Node: 4, down, Subscriber bitmask 00
121219 18:19:46 [Note] NDB Binlog: Node: 5, down, Subscriber bitmask 00
121219 18:19:46 [Note] NDB Binlog: cluster failure for ./mysql/ndb_schema at epoch 8955534/13.
121219 18:19:46 [Note] NDB Binlog: cluster failure for ./mysql/ndb_apply_status at epoch 8955534/13.
121219 18:19:46 [Note] Restarting Cluster Binlog
121219 18:19:46 [ERROR] /usr/sbin/mysqld: Incorrect information in file: './ndb/fl_state.frm'
121219 18:19:47 [ERROR] /usr/sbin/mysqld: Incorrect information in file: './ndb/fl_state.frm'
121219 18:19:47 [ERROR] /usr/sbin/mysqld: Incorrect information in file: './ndb/fl_state.frm'
121219 18:19:47 [ERROR] /usr/sbin/mysqld: Incorrect information in file: './ndb/fl_state.frm'
121219 18:19:47 [ERROR] /usr/sbin/mysqld: Incorrect information in file: './ndb/fl_state.frm'
121219 18:19:47 [ERROR] /usr/sbin/mysqld: Incorrect information in file: './ndb/fl_state.frm'
121219 18:19:47 [Note] table './ndb/fl_state' opened read only
121219 18:19:47 [Note] table './ndb/fl_state' opened read only
121219 18:19:47 [Note] table './ndb/fl_state' opened read only
121219 18:19:47 [Note] table './ndb/fl_state' opened read only
121219 18:19:47 [Note] table './ndb/fl_state' opened read only
121219 18:19:47 [Note] table './ndb/fl_state' opened read only
121219 18:19:47 [Note] NDB Binlog: DISCOVER TABLE Event: REPL$mysql/ndb_schema
121219 18:19:47 [Note] NDB Binlog: logging ./mysql/ndb_schema (UPDATED,USE_WRITE)
121219 18:19:47 [Note] NDB Binlog: DISCOVER TABLE Event: REPL$mysql/ndb_apply_status
121219 18:19:48 [Note] NDB Binlog: logging ./mysql/ndb_apply_status (UPDATED,USE_WRITE)
121219 18:19:48 [Note] NDB: Cleaning stray tables from database 'information_schema'
121219 18:19:48 [Note] NDB: Cleaning stray tables from database 'Web_Members'
121219 18:19:48 [Note] NDB: Cleaning stray tables from database 'cluster'
121219 18:19:48 [Note] NDB: Cleaning stray tables from database 'ndb'
121219 18:19:48 [Note] NDB: Cleaning stray tables from database 'ndbinfo'
121219 18:19:48 [Note] NDB: Cleaning stray tables from database 'performance_schema'
121219 18:19:48 [Note] NDB: Cleaning stray tables from database 'test'
2012-12-19 18:19:48 [NdbApi] INFO -- Flushing incomplete GCI:s < 8955546/3
2012-12-19 18:19:48 [NdbApi] INFO -- Flushing incomplete GCI:s < 8955546/3
121219 18:19:48 [Note] NDB Binlog: starting log at epoch 8955546/3
121219 18:19:48 [Note] NDB Binlog: ndb tables writable
121219 18:19:48 [Note] NDB Binlog: Node: 4, subscribe from node 50, Subscriber bitmask 400000
121219 18:19:48 [Note] NDB Binlog: Node: 4, subscribe from node 51, Subscriber bitmask c00000
121219 18:19:48 [Note] NDB Binlog: Node: 4, subscribe from node 62, Subscriber bitmask 400c00000
121219 18:19:48 [Note] NDB Binlog: Node: 4, subscribe from node 57, Subscriber bitmask 420c00000
121219 18:19:48 [Note] NDB Binlog: Node: 4, subscribe from node 53, Subscriber bitmask 422c00000
121219 18:19:48 [Note] NDB Binlog: Node: 4, subscribe from node 59, Subscriber bitmask 4a2c00000
121219 18:19:48 [Note] NDB Binlog: Node: 4, subscribe from node 55, Subscriber bitmask 4aac00000
121219 18:19:48 [Note] NDB Binlog: Node: 4, subscribe from node 52, Subscriber bitmask 4abc00000
121219 18:19:48 [Note] NDB Binlog: Node: 4, subscribe from node 54, Subscriber bitmask 4afc00000
121219 18:19:48 [Note] NDB Binlog: Node: 4, subscribe from node 58, Subscriber bitmask 4efc00000
121219 18:19:48 [Note] NDB Binlog: Node: 5, subscribe from node 50, Subscriber bitmask 400000
121219 18:19:48 [Note] NDB Binlog: Node: 5, subscribe from node 51, Subscriber bitmask c00000
121219 18:19:48 [Note] NDB Binlog: Node: 5, subscribe from node 62, Subscriber bitmask 400c00000
121219 18:19:48 [Note] NDB Binlog: Node: 5, subscribe from node 57, Subscriber bitmask 420c00000
121219 18:19:48 [Note] NDB Binlog: Node: 5, subscribe from node 53, Subscriber bitmask 422c00000
121219 18:19:48 [Note] NDB Binlog: Node: 5, subscribe from node 59, Subscriber bitmask 4a2c00000
121219 18:19:48 [Note] NDB Binlog: Node: 5, subscribe from node 55, Subscriber bitmask 4aac00000
121219 18:19:48 [Note] NDB Binlog: Node: 5, subscribe from node 58, Subscriber bitmask 4eac00000
121219 18:19:48 [Note] NDB Binlog: Node: 5, subscribe from node 54, Subscriber bitmask 4eec00000
121219 18:19:48 [Note] NDB Binlog: Node: 5, subscribe from node 52, Subscriber bitmask 4efc00000

mgmd:

(Node 56 = mysqld above)

2012-12-19 18:19:21 [MgmtSrvr] WARNING -- Node 5: Node 56 missed heartbeat 2
2012-12-19 18:19:22 [MgmtSrvr] WARNING -- Node 4: Node 56 missed heartbeat 2
2012-12-19 18:19:24 [MgmtSrvr] WARNING -- Node 5: Node 56 missed heartbeat 3
2012-12-19 18:19:24 [MgmtSrvr] WARNING -- Node 4: Disconnecting node 56 because it has exceeded MaxBufferedEpochs (100 > 100), epoch 8955529/10
2012-12-19 18:19:24 [MgmtSrvr] INFO -- Node 4: Communication to Node 56 closed
2012-12-19 18:19:24 [MgmtSrvr] INFO -- Node 5: Communication to Node 56 closed
2012-12-19 18:19:24 [MgmtSrvr] WARNING -- Node 5: Disconnecting node 56 because it has exceeded MaxBufferedEpochs (100 > 100), epoch 8955529/10
2012-12-19 18:19:25 [MgmtSrvr] ALERT -- Node 4: Node 56 Disconnected
2012-12-19 18:19:25 [MgmtSrvr] ALERT -- Node 5: Node 56 Disconnected
2012-12-19 18:19:28 [MgmtSrvr] INFO -- Node 5: Communication to Node 56 opened
2012-12-19 18:19:28 [MgmtSrvr] INFO -- Node 4: Communication to Node 56 opened
2012-12-19 18:19:33 [MgmtSrvr] INFO -- Node 5: Data usage is 0%(214 32K pages of total 327680)
2012-12-19 18:19:33 [MgmtSrvr] INFO -- Node 5: Index usage is 0%(50 8K pages of total 1310752)
2012-12-19 18:19:33 [MgmtSrvr] INFO -- Node 4: Data usage is 0%(214 32K pages of total 327680)
2012-12-19 18:19:33 [MgmtSrvr] INFO -- Node 4: Index usage is 0%(50 8K pages of total 1310752)
2012-12-19 18:19:37 [MgmtSrvr] INFO -- Node 4: Local checkpoint 145627 started. Keep GCI = 8955509 oldest restorable GCI = 8955512
2012-12-19 18:19:45 [MgmtSrvr] INFO -- Node 4: Local checkpoint 145627 completed
2012-12-19 18:19:47 [MgmtSrvr] INFO -- Node 4: Node 56 Connected
2012-12-19 18:19:47 [MgmtSrvr] INFO -- Node 5: Node 56 Connected
2012-12-19 18:19:47 [MgmtSrvr] INFO -- Node 4: Node 56: API mysql-5.5.27 ndb-7.2.8
2012-12-19 18:19:47 [MgmtSrvr] INFO -- Node 5: Node 56: API mysql-5.5.27 ndb-7.2.8

Hi,
I'm searching for experiences using a myIsam master which replicates (using row replication) to a NDB Cluster Slave.

It worked but suddenly, after a load with dump, ndb cluster switched its engine to MyIsam.

There is a way to lock slave engine to ndb?

Thanks,
Guido

I'm trying to set up a cluster on Amazon EC2. Two of my instances will serve as Management/SQL nodes, while two others will serve as data nodes only (I've had this working in my development environment for 7 months).

No matter what I do, the SQL nodes start but won't connect. The output of ndb_mgm -e show is always:

Cluster Configuration
---------------------
[ndbd(NDB)] 2 node(s)
id=3 @10.60.83.18 (mysql-5.5.27 ndb-7.2.8, Nodegroup: 0, Master)
id=4 @10.6.99.242 (mysql-5.5.27 ndb-7.2.8, Nodegroup: 0, Master)

[ndb_mgmd(MGM)] 2 node(s)
id=1 @10.245.70.220 (mysql-5.5.27 ndb-7.2.8)
id=2 @10.156.145.133 (mysql-5.5.27 ndb-7.2.8)

[mysqld(API)] 2 node(s)
id=5 (not connected, accepting connect from 10.245.70.220)
id=6 (not connected, accepting connect from 10.156.145.133)

I've been Googling this and trying different things for about the last 12 hours straight, so I'm posting here and hoping for a Christmas miracle. Does anyone have any idea what might be causing this?

Hi,

I've done a few changes to my config.ini how can I find out if this changes are active ? specifically I need to know if RealtimeScheduler is active or not
I've tried ndb_config

but this is all that I can find

RealtimeScheduler (Boolean value)
If yes, then NDB Cluster threads will be scheduled as real-time threads
Default: N (Legal values: Y, N)


I'm using 5.1.47-ndb-7.1.5-cluster-gpl
and we don't have mysql cluster manager

I am running mysql cluster 7.2.5 with one mgm node, two sql nodes, and two data nodes. When running ndb_restore, I can restore to the first data node, but I am unable to restore to the second node.

When restoring to the first data node I run: ndb_restore -n 2 -m -b 28
And I receive a listing of all the successfully restored tables and successfully created indexes.

But on the second data node, I never receive the "Connected to ndb!!" message followed by a listing of the successfully restored tables.

Instead I receive this:

[mysql@SERVER BACKUP-28]$ ndb_restore -n 3 -b 28
Nodeid = 3
Backup Id = 28
backup path = ./
Opening file './BACKUP-28.3.ctl'
File size 92160 bytes
Backup version in files: ndb-6.3.11 ndb version: mysql-5.5.20 ndb-7.2.5
Stop GCP of Backup: 10938390

NDBT_ProgramExit: 0 - OK

Any help would be greatly appreciated. Thank you.

Hi ,

I have very urgent questions related to backup/restores of databases in a node of ndbcluster environment.

I used mysqldump to backup all the databases. it got successful. Now when i try to restores those db dumps, i see the error in restoring 'mysql' system database.

here are the results:

[root@IfxLinux mysql-cluster]# mysql < /tmp/my.sql
ERROR 1036 (HY000) at line 386: Table 'ndb_apply_status' is read only

error says 'ndb_apply_status' table is read only and restore halts.

even dropping 'mysql' database shows error:
system command = /usr/bin/mysql -u backup --socket=/var/lib/mysql-cluster/mysql.sock -e 'FLUSH PRIVILEGES;drop database `mysql`;' out err = ERROR 1051 (42S02) at line 1: Unknown table 'ndb_apply_status'.


Please help and advise how to resolve this. We are able to restore all other databases apart from 'mysql' database from a dump.

#Note: we don't want to use start backup and ndb_restore programs.

Help appreciated.

i am new to mysql clusters and still reading and learning. i have a question for which i couldn't find a particular answer on forums.

what happens when data node disconnects from the network? does cluster database still stays in memory of the data node and can be accessed for quering or is that point (data node) "dead" until connection is restored?

thx

Hello,

I have got following Error while installing the MySQL Cluster Server 7.2.8 (MySQL-Cluster-server-gpl-7.2.8-1.rhel5.x86_64.rpm) after removing old RPMs of MySQL Cluster 7.1.18

Preparing packages for installation...
ls: /var/lib/mysql/*.err: No such file or directory
ls: /var/lib/mysql/*.err: No such file or directory
MySQL-Cluster-server-gpl-7.2.8-1.rhel5
/var/tmp/rpm-tmp.13550: line 23: [: /var/lib/mysql: binary operator expected
/var/tmp/rpm-tmp.13550: line 24: [: /var/lib/mysql: binary operator expected
/var/tmp/rpm-tmp.13550: line 30: [: /var/lib/mysql: binary operator expected
/var/tmp/rpm-tmp.13550: line 35: /var/lib/mysql
/var/lib/mysql/RPM_UPGRADE_MARKER: No such file or directory
chmod: cannot access `/var/lib/mysql/mysql': No such file or directory
/var/tmp/rpm-tmp.13550: line 140: /var/lib/mysql
/var/lib/mysql/RPM_UPGRADE_MARKER: No such file or directory
/var/tmp/rpm-tmp.13550: line 141: /var/lib/mysql
/var/lib/mysql/RPM_UPGRADE_MARKER: No such file or directory
/var/tmp/rpm-tmp.13550: line 142: /var/lib/mysql
/var/lib/mysql/RPM_UPGRADE_MARKER: No such file or directory
/var/tmp/rpm-tmp.13550: line 144: /var/lib/mysql
/var/lib/mysql/RPM_UPGRADE_HISTORY: No such file or directory
mv: target `/var/lib/mysql/RPM_UPGRADE_MARKER-LAST' is not a directory
error: %post(MySQL-Cluster-server-gpl-7.2.8-1.rhel5.x86_64) scriptlet failed, exit status 1

Though my Functional Setup (1MGM, 1DBD , 1 MYSQLD) came up successfully, but want to know the reason and possible consequences of the same.

Thanks in Advance.

I am trying to setup replication with a single replication channel between two MySQL clusters. I have followed the documentation from mysql.com a couple of times, and can not seem to get it working properly.

The problem I'm having is that queries made to an SQL node that is not configured as the Master is not replicating any INSERT,UPDATE or DELETE queries for NDBCLUSTER tables, however when I INSERT,UPDATE or DELETE a row on the SQL node that is the master, it replicates to the other cluster fine.

I know that replication is setup because if I run a CREATE DATABASE query on any SQL Node in my Master cluster, it replicates fine to the Slave cluster. Likewise, if I run a CREATE TABLE query on any SQL node to create a NDBCLUSTER table in my new database, it too replicates to the Slave cluster fine.

The way I read the documentation and how I have understood cluster replication to work is that you could have 10 API nodes in your cluster, with one of those nodes configured for and dedicated to replication as the Master. So if an INSERT,UPDATE or DELETE query occurs on any of the 10 API nodes in the cluster, then the replication master and the NDB binlog will pick this up allowing slaves to then replicate. This does not seem to be working for me.

Is my understanding correct?

Here's how I've set things up:

All servers are VMs running 64-bit CentOS 6.2 and MySQL Cluster 7.2.8 x86_64.

Cluster 1 (Master Cluster):
1 x ndb_mgmd Management Node (Server 1A - IP: 10.14.64.10)
2 x mysqld API nodes (Servers 1B and 1C - IPs: 10.14.64.11 and 10.14.64.12)
2 x ndbd Data nodes (Servers 1D and 1E - IPs: 10.14.64.13 and 10.14.64.14)

Cluster 2 (Slave Cluster):
1 x ndb_mgmd Management Node (Server 2A - IP: 10.13.64.10)
2 x mysqld API nodes (Servers 2B and 2C - IPs: 10.13.64.11 and 10.13.64.12)
2 x ndbd Data nodes (Servers 2D and 2E - IPs: 10.13.64.13 and 10.13.64.14)

Here is a pretty picture to assist in explaining:

http://cpjon.es/wp-content/uploads/2013/01/mysql-cluster-replication.png

I have replication setup so that Server 1C is the Replication Master and Server 2C is the Replication Slave.

Here is the /etc/my.cnf file for Server 1C (being the MASTER):
====================================================
[mysqld]
ndbcluster
ndb-connectstring=10.14.64.10
log-bin
binlog-format=ROW
server-id=10

[mysql_cluster]
ndb-connectstring=10.14.64.10
====================================================

Here is the /etc/my.cnf file for Server 2C (being the SLAVE):
====================================================
[mysqld]
ndbcluster
ndb-connectstring=10.13.64.10
server-id=11

[mysql_cluster]
ndb-connectstring=10.13.64.10
====================================================

When I run the query CREATE DATABASE test_db1; on either SQL Server 1B or 1C, it is replicated instantly to Cluster 2, and I can see database test_db1 on SQL Servers 2B and 2C.

When I run the following CREATE TABLE query:

CREATE TABLE `City` (
`ID` int(11) NOT NULL auto_increment,
`Name` char(35) NOT NULL default '',
`CountryCode` char(3) NOT NULL default '',
`District` char(20) NOT NULL default '',
`Population` int(11) NOT NULL default '0',
PRIMARY KEY (`ID`)
) ENGINE=NDBCLUSTER DEFAULT CHARSET=latin1;

on either SQL Server 1B or 1C, it is replicated instantly to Cluster 2, and I can see my newly created table City on SQL Servers 2B and 2C.

However, here is the problem - when I run the following query on Server 1B:

INSERT INTO City VALUES (1,'Melbourne','AUS','Victoria',4100000);
I can see the entry on all SQL nodes in Cluster 1 (ie. Servers 1B and 1C), but can not see the entry on any SQL nodes in Cluster 2 (ie. Server 2B and 2C).

If I run a similar query on Server 1C:

INSERT INTO City VALUES (2,'Sydney','AUS','New South Wales',4600000);
I can see the entry on all SQL nodes in Cluster 1 AND Cluster 2 - so on all four SQL nodes - which is what I expect.

I must be missing something really simple. I have tried blowing away these servers and starting again on numerous occations - and can't seem to get any INSERT, UPDATE or DELETE queries on Server 1B going to my secondary cluster.

Any pointers or assistance would be greatly appreciated (and sorry for the long winded post)

Regards,
Chris.

Using ClusterJ api to load data and after about a million or so rows the cluster shuts down.
Using mysql cluster 7.2.7 installed.

ndb_error_reporter output can be found at:
https://docs.google.com/open?id=0ByPsQoJxX4VXSEd4THR3OHJxbmM

Here are the last few log entries from when the cluster shuts down.

2013-01-04 12:20:30 [MgmtSrvr] ALERT -- Node 3: Forced node shutdown completed. Caused by error 2301: 'Assertion(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'.
2013-01-04 12:20:30 [MgmtSrvr] INFO -- Node 2: Local checkpoint 22 started. Keep GCI = 912 oldest restorable GCI = 923
2013-01-04 12:20:30 [MgmtSrvr] ALERT -- Node 2: Node 3 Disconnected
2013-01-04 12:20:34 [MgmtSrvr] INFO -- Node 2: Communication to Node 3 opened
2013-01-04 12:20:34 [MgmtSrvr] INFO -- Node 5: Communication to Node 3 opened
2013-01-04 12:20:34 [MgmtSrvr] INFO -- Node 4: Communication to Node 3 opened
2013-01-04 12:20:34 [MgmtSrvr] INFO -- Node 14: Communication to Node 3 opened
2013-01-04 12:20:39 [MgmtSrvr] INFO -- Node 14: Data usage is 0%(2968 32K pages of total 1424832)
2013-01-04 12:20:39 [MgmtSrvr] INFO -- Node 14: Index usage is 0%(1077 8K pages of total 712576)
2013-01-04 12:20:40 [MgmtSrvr] INFO -- Node 2: Data usage is 0%(3098 32K pages of total 1424832)
2013-01-04 12:20:40 [MgmtSrvr] INFO -- Node 2: Index usage is 0%(1096 8K pages of total 712576)
2013-01-04 12:20:43 [MgmtSrvr] INFO -- Node 2: Local checkpoint 22 completed
2013-01-04 12:20:43 [MgmtSrvr] INFO -- Node 5: Data usage is 0%(2986 32K pages of total 1424832)
2013-01-04 12:20:43 [MgmtSrvr] INFO -- Node 5: Index usage is 0%(1079 8K pages of total 712576)
2013-01-04 12:20:44 [MgmtSrvr] INFO -- Node 4: Data usage is 0%(2987 32K pages of total 1424832)
2013-01-04 12:20:44 [MgmtSrvr] INFO -- Node 4: Index usage is 0%(1079 8K pages of total 712576)
2013-01-04 12:20:44 [MgmtSrvr] INFO -- Node 2: Local checkpoint 23 started. Keep GCI = 924 oldest restorable GCI = 932
2013-01-04 12:20:57 [MgmtSrvr] INFO -- Node 2: Local checkpoint 23 completed
2013-01-04 12:20:58 [MgmtSrvr] INFO -- Node 2: Local checkpoint 24 started. Keep GCI = 935 oldest restorable GCI = 943
2013-01-04 12:21:14 [MgmtSrvr] INFO -- Node 4: Data usage is 0%(3323 32K pages of total 1424832)
2013-01-04 12:21:14 [MgmtSrvr] INFO -- Node 4: Index usage is 0%(1194 8K pages of total 712576)
2013-01-04 12:21:17 [MgmtSrvr] ALERT -- Node 14: Forced node shutdown completed. Occured during startphase 0. Initiated by signal 11.
2013-01-04 12:21:17 [MgmtSrvr] ALERT -- Node 1: Node 14 Disconnected
2013-01-04 12:21:17 [MgmtSrvr] ALERT -- Node 4: Node 14 Disconnected
2013-01-04 12:21:19 [MgmtSrvr] ALERT -- Node 1: Node 2 Disconnected
2013-01-04 12:21:19 [MgmtSrvr] ALERT -- Node 2: Forced node shutdown completed. Caused by error 2301: 'Assertion(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'.
2013-01-04 12:21:19 [MgmtSrvr] ALERT -- Node 4: Node 2 Disconnected
2013-01-04 12:21:21 [MgmtSrvr] ALERT -- Node 5: Forced node shutdown completed. Caused by error 2341: 'Internal program error (failed ndbrequire)(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'.
2013-01-04 12:21:21 [MgmtSrvr] ALERT -- Node 4: Node 5 Disconnected
2013-01-04 12:21:21 [MgmtSrvr] ALERT -- Node 1: Node 5 Disconnected
2013-01-04 12:21:31 [MgmtSrvr] ALERT -- Node 4: Forced node shutdown completed. Caused by error 2305: 'Node lost connection to other nodes and can not form a unpartitioned cluster, please investigate if there are error(s) on other node(s)(Arbitration error). Temporary error, restart node'.
2013-01-04 12:21:31 [MgmtSrvr] ALERT -- Node 1: Node 4 Disconnected