So my first experience with MySQL cluster/NDB has been far less than stellar.

I have a corrupt database, which has never happened with InnoDB.

Set up and ran a cluster, low volume, NoCopies=2

1 x NDB + mysql on metal where all clients connected
1 x NDB + mysql on Oracle VirtualBox
1 x mnmt on secondary Oracle VirtualBox

During the course of operation, I got some table read errors, Err 233. (No text error message associated.) Looked it up, very suspicious that I was hitting the 32k MaxNoOfConcurrentOperations limit, but moved it up anyways to 65535.

Eventually ran out of disk on the NDB+mysql VM. Shut everything down, restarted with another VM disk attached for data and bumped memory alloc while it was down.

Still getting MaxNoOfConcurrentOperations-type errors, but fewer, even at 65535. This is dumb because we're sub 20 (TWENTY!) TPS with almost all of them being reads. No way it should be hitting 32k concurrent, much less 64k. These showed up on the bare-metal box, not the VM box.

Looking at the database, it's showing corruption ... it looks like some of transactions were completely replayed... the UPDATE calls which used "col = col + x" are all skewed. (Note: I did not tell it to use binlog or =row.)

1) Why would I possibly be getting Err: 233 off such a stupid low number of twenty TPS on the bare-metal box? I could see the VM potentially having issues, but it was literally just a replication target in the cluster.

2) Why would I end up with getting replayed statement transactions? How can I even start to trace this? How does this even happen? And importantly, how can I ensure that this never (ever) happens again?


Thanks ... hopefully someone can shed some light on this!

I am student and I am writing Master of Science paper about in-memory databases. I want to use MySql Cluster for benchmarks in my work.
Can I use MySql Cluster free of charge for educational purposes or do I need to pay for it?

I have successfully installed the Cluster with 5 nodes (2 Data, 1 Mgmt and 2 SQL Nodes). While testing the cluster I have hit on one scenario where I am stuck and cannot make it to work. Here is the scenario:

While all nodes in the cluster are working as a part of my test I shut down one of the SQL A node and while this node is offline, I drop a database which is part of the cluster databases on SQL Node B. When I bring up the offline SQL Node A and rejoin to the cluster the dropped database still shows up. It should sync with the old cluster databases meaning the dropped database when the SQL Node A was offline should be removed from the cluster and should not show up on SQL Node A. This is a real scenario that can happen.

Also, I am searching for MySQL Cluster Test document which describes these scenarios and cannot seem to find it.

Any Help will be greatly appreciated.

Thanks
Lal

I am new to MySql Cluster.
I have evaluated MySQL Cluster Quick Start Guide.
In this guide in "Run" section are commands to run two data nodes on the same host. What is the point of such operation?
If I want run cluster for test purposes (do not need failure safeness) with one host only do I need two data nodes or only one data node can work?

My mysql cluster data nodes data usage increased to 95%. Then i tried to delete 500000 rows and my data usage of nodes started increasing initially and reached to 99% and above and TABLE FULL error is encountered. Please can anyone tell me why data usage of existing nodes increased when deleting rows? What is the internal working of mysql cluster when deleting rows?

My MaxNoOfConcurrentOperations parameter in config.ini 600000 and it is not due to this parameter.

In my mysql cluster i have 1 million rows and my disk data tables are using all the disk usage available to them ;i.e; free extents are 0 . Now i deleted 50K rows from ap_perf_tbl but still it show free extents as 0 but i am able to insert into that table . But when i tried to insert into another table associated with that tablespace it says TABLE FULL error. I am not able to use those freed up extents for another table . Is there any way to use those freed extents from one table to be used by another table ?

I deleted 500000 rows in my mysql cluster table having 12468000 rows and then i tried to OPTIMIZE TABLE so that all the freed up data memory can be used by another table also, but after optimizing table the data usage increased instead of decreasing it. Please explain how OPTIMIZE TABLE works and why data usage increased.

DATA USAGE BEFORE OPTIMIZING TABLE:

Node 2: Data usage is 71%(7348 32K pages of total 10240)
Node 2: Index usage is 4%(4382 8K pages of total 102432)
Node 3: Data usage is 69%(7167 32K pages of total 10240)
Node 3: Index usage is 4%(4291 8K pages of total 102432)
Node 4: Data usage is 70%(7229 32K pages of total 10240)
Node 4: Index usage is 4%(4312 8K pages of total 102432)
Node 5: Data usage is 67%(6899 32K pages of total 10240)
Node 5: Index usage is 4%(4201 8K pages of total 102432)


DATA USAGE AFTER OPTIMIZING TABLE:


Node 2: Data usage is 74%(7589 32K pages of total 10240)
Node 2: Index usage is 4%(4382 8K pages of total 102432)
Node 3: Data usage is 72%(7392 32K pages of total 10240)
Node 3: Index usage is 4%(4291 8K pages of total 102432)
Node 4: Data usage is 72%(7461 32K pages of total 10240)
Node 4: Index usage is 4%(4312 8K pages of total 102432)
Node 5: Data usage is 69%(7117 32K pages of total 10240)
Node 5: Index usage is 4%(4201 8K pages of total 102432)

Hi,
I am new to the mysql. I am trying to setup mysql cluster. Could you please tell me whether I need to download ONLY "mysql-cluster-gpl-7.2.10-win32" or I should download the MySQL installer "mysql-installer-community-5.6.10.1.msi" too? I don't find any docs which will tell step by step guide to install mysql cluster in windows 7 or even any flavor of windows. All that I got is install mysql (*.msi) on windows.
I followed the steps that are mentioned in the doc "MySQL Cluster Quick Start Guide – Windows" but not able to connect as I don't know what is the root password (mysql -h 127.0.0.1 -P 3306 -u root -p) I should provide as no where it asked me to setup any root password. So I am bit doubt whether I should install mysql installer prior to cluster setup.
Could you please tell me if any docs exist so that I can follow the cluster setup?
Regards,
Saroj

Hi,I'm trying mysql cluster to use in a project for our customer.

The configuration I setted up is the following:
Hardware S.O. equipment
Server1 - S.O. Windows 2008 R2 64bit
Server2 - S.O. Windows 2008 R2 64bit
Client1 - S.O. Windows 7 64bit
Client2 - S.O. Windows 7 64bit

UserClient2 - S.O. Windows 7 64bit
UserClient3 - S.O. Windows 7 64bit
UserClient4 - S.O. Windows 7 64bit
UserClient5 - S.O. Windows 7 64bit
UserClientXX - S.O. Windows 7 64bit (where XX is less than 15)

My Sql Cluster Configuration
Server1 - act as cluster data node 1
Server2 - act as cluster data node 2
Server1 - act as mysql node 1 (from now MYSQL-D1)
Server2 - act as mysql node 2 (from now MYSQL-D2)

Client1 - act as mysql cluster management node 1
Client2 - act as mysql cluster management node 2

Client application configuration
The following clients (from now CLIENTAPP):
Client1, Client2, UserClient2, .., UserClientXX
have a Microsoft .Net application that interact with mysql cluster database.

After a while the My sql cluster configuration appears to work correctly and any of CLIENTAPP it's able to connect to MYSQL-D1 or MYSQL-D2 (this is possibile by changing the connection string of .NET client application and pointing to IP of MYSQL-D1 or to the IP of MYSQL-D2).

In a perfect world CLIENTAPP automatically it's able to connect to the right server (MYSQL-D1 or MYSQL-D2), pratically:
- if MYSQL-D1 it's down CLIENTAPP connect to MYSQL-D2
- if MYSQL-D2 it's down CLIENTAPP connect to MYSQL-D1

How .NET client application can interact with mysql cluster?

The best solutions I have found for my scenario are the followings:
1. use mysql proxy but this is in a alpha stage and I can't use it in production enviroment
2. install cluster data node on each CLIENTAPP
3. use a connection string that support multiple servers (I don't know if it's supported by mysql .NET connector)
4. rewrite my application to interact with both MYSQL-DX node (what I want to avoid)

Is there a right solutions? Which one is the best one?

Thank you

Hello.

I've got problem with cluster getting following error:
General error: 1297 Got temporary error 899 'Rowid already allocated' from NDBCLUSTER

I'm running cluster (ndb 7.2.10) with two machines - one data node and one mysql node on each machine, with diskless turned on.
The problem starts occuring under heavy load - few hundreds inserts and updates per second on each machine. What is strange, after it starts occuring it doesn't stop even when load gets very small.
Only inserts cause this problem.
Queries sometimes cause this error and sometimes don't (ie. every second 50% of queries fail with rowid error).
Error stops appearing for inserts into table that was just truncated.
Some tables have primary key with auto_increment, others don't - insert into all can cause error.
Insert into biggest table (about 100M inserts a day) and few other tables don't cause error (didn't find any correlation yet).
Error stops occuring after restarting/stopping one of data nodes.

I've got no idea how to approach this problem - any suggestions? Or maybe it's a known problem which can be easily fixed? :)

hi~

i wonder

2 NDB node config.ini index memory + data memory set 45G

so NDB3 NDB4 memory usage different

why different ?




NDB 3
[mysql@mysql ]$ free -m
total used free shared buffers cached
Mem: 64454 62768 1686 0 168 25767
-/+ buffers/cache: 36832 27622
Swap: 32765 25566 7198

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
23551 mysql 15 0 59.3g 34g 3832 S 26.2 54.4 27:27.12 ndbmtd


NDB 4

[mysql@mysql]$ free -m
total used free shared buffers cached
Mem: 64454 64285 169 0 94 17796
-/+ buffers/cache: 46394 18060
Swap: 32765 13583 19181
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
15113 mysql 15 0 59.2g 44g 3304 S 25.2 70.6 18:46.54 ndbmtd

Hi MySQL/Oracle,

A while ago I posted that my mysqld was crashing about once a week. This is still the case. I wonder whether any member may have an idea why?

Using 7.2.10 on Linux 3.1.10-1.9.

Only a single NDB table at abut 50 DML/sec from various clients.
Table with 227 columns, 5 keys, 6567 rows, and average row length of 884 bytes.

Table schema, information schema report and MySQL crash stack trace to follow.

Because Cluster keep crashing, it's giving cluster a bad name in this company. Which is a shame, as I want this to work. If anybody has any ideas, please let me know.

Regards, Ben.

-----------------------------------------------------

Crash log:

05:54:07 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

key_buffer_size=8388608
read_buffer_size=5242880
max_used_connections=37
max_threads=200
thread_count=34
connection_count=34
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2058485 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x6ca3750
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7fd9081b9e98 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x35)[0x81cb65]
/usr/sbin/mysqld(handle_fatal_signal+0x403)[0x6e8f33]
/lib64/libpthread.so.0(+0xfd00)[0x7fd99e25fd00]
/usr/sbin/mysqld(_Z15ha_commit_transP3THDb+0x158)[0x6ecc08]
/usr/sbin/mysqld(_Z17trans_commit_stmtP3THD+0x1d)[0x69ad8d]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x4c4)[0x5ee4b4]
/usr/sbin/mysqld(_ZN21Execute_sql_statement19execute_server_codeEP3THD+0xf3)[0x5fea43]
/usr/sbin/mysqld(_ZN18Prepared_statement23execute_server_runnableEP15Server_runnable+0x159)[0x5fe2b9]
/usr/sbin/mysqld(_ZN13Ed_connection14execute_directEP15Server_runnable+0xc9)[0x5ffdf9]
/usr/sbin/mysqld(_ZN13Ed_connection14execute_directE19st_mysql_lex_string+0x2d)[0x5ffecd]
/usr/sbin/mysqld(_ZN20Ndb_local_connection13execute_queryE19st_mysql_lex_stringPKjPK10Suppressor+0x6b)[0x9db87b]
/usr/sbin/mysqld(_ZN20Ndb_local_connection17execute_query_isoE19st_mysql_lex_stringPKjPK10Suppressor+0xc8)[0x9dba48]
/usr/sbin/mysqld(_ZN20Ndb_local_connection11delete_rowsEPKcmS1_mbz+0x182)[0x9dc002]
/usr/sbin/mysqld[0x9cb70f]
/usr/sbin/mysqld[0x6ea2bf]
/usr/sbin/mysqld(_Z26ha_binlog_index_purge_fileP3THDPKc+0x24)[0x6ea304]
/usr/sbin/mysqld(_ZN13MYSQL_BIN_LOG17purge_index_entryEP3THDPyb+0x3c9)[0x77dc39]
/usr/sbin/mysqld(_ZN13MYSQL_BIN_LOG10purge_logsEPKcbbbPy+0x1a9)[0x77f189]
/usr/sbin/mysqld(_ZN13MYSQL_BIN_LOG22purge_logs_before_dateEl+0x320)[0x77fda0]
/usr/sbin/mysqld(_ZN13MYSQL_BIN_LOG5writeEP3THDP11st_io_cacheP9Log_eventb+0x418)[0x783b58]
/usr/sbin/mysqld[0x78443c]
/usr/sbin/mysqld[0x7845e3]
/usr/sbin/mysqld(_Z19ha_commit_one_phaseP3THDb+0x87)[0x6ec9e7]
/usr/sbin/mysqld(_Z15ha_commit_transP3THDb+0x1d1)[0x6ecc81]
/usr/sbin/mysqld(_Z17trans_commit_stmtP3THD+0x1d)[0x69ad8d]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x4c4)[0x5ee4b4]
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x18a)[0x5f285a]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x1331)[0x5f4411]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x167)[0x68d657]
/usr/sbin/mysqld(handle_one_connection+0x54)[0x68d6c4]
/lib64/libpthread.so.0(+0x7f05)[0x7fd99e257f05]
/lib64/libc.so.6(clone+0x6d)[0x7fd99d4fb10d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7fd884071bf0): is an invalid pointer
Connection ID (thread ID): 24
Status: NOT_KILLED


-----------------------------------------------------

Table Schema:

CREATE TABLE `fl_state` (
`terminal_id` varchar(32) NOT NULL,
`ip_addr` varchar(15) NOT NULL DEFAULT '',
`server_name` varchar(40) NOT NULL DEFAULT '',
`server_pid` int(11) NOT NULL DEFAULT '0',
`version_daemon` varchar(20) NOT NULL DEFAULT '',
`version_software` varchar(20) NOT NULL DEFAULT '',
`version_protocol` varchar(20) NOT NULL DEFAULT '',
`connected` timestamp NULL DEFAULT NULL,
`tally` int(11) NOT NULL DEFAULT '0',
`tacho_type` varchar(20) DEFAULT NULL,
`comp_code` varchar(8) DEFAULT NULL,
`veh_code` varchar(16) DEFAULT NULL,
`veh_rtid` int(11) DEFAULT NULL,
`mem_code` int(11) DEFAULT NULL,
`terminal_type` varchar(20) DEFAULT NULL,
`terminal_type_sub` varchar(20) DEFAULT NULL,
`terminal_type_sub_from_veh` varchar(20) DEFAULT NULL,
`service_code` varchar(16) DEFAULT NULL,
`service_sub_code` varchar(8) DEFAULT NULL,
`entity_type` enum('driven','trailer','member') NOT NULL DEFAULT 'driven',
`site_no` int(11) DEFAULT NULL,
`reset_required` tinyint(1) NOT NULL DEFAULT '0',
`tag_master_veh_rtid` int(11) DEFAULT NULL,
`tag_master_veh_rtid_prev` int(11) DEFAULT NULL,
`tag_master_terminal_id` varchar(32) DEFAULT NULL,
`tag_slave_recorded` tinyint(1) NOT NULL DEFAULT '0',
`tag_slave_count` int(11) NOT NULL DEFAULT '0',
`tag_state_pending` enum('trip','timeout','next_valid','not') NOT NULL DEFAULT 'not',
`tag_state_pending_t` timestamp NULL DEFAULT NULL,
`tag_state_pending_lat` decimal(9,6) DEFAULT NULL,
`tag_state_pending_lon` decimal(9,6) DEFAULT NULL,
`tag_state_pending_km_h` int(11) DEFAULT NULL,
`tag_state_pending_id` int(11) DEFAULT NULL,
`tag_state_timeout_start` timestamp NULL DEFAULT NULL,
`tag_state_timeout_t` int(11) DEFAULT NULL,
`tag_state_pending_trip` enum('wait_primed','wait_next','found_trip','not') NOT NULL DEFAULT 'not',
`tag_slave_last_drop` timestamp NULL DEFAULT NULL,
`tag_slave_last_hookup` timestamp NULL DEFAULT NULL,
`fl_story_rec` bigint(20) unsigned DEFAULT NULL,
`mo_t` timestamp NOT NULL DEFAULT '2000-01-01 00:00:00',
`mo_odo` decimal(11,3) DEFAULT NULL,
`mo_fuel` decimal(10,2) DEFAULT NULL,
`md_t` timestamp NOT NULL DEFAULT '2000-01-01 00:00:00',
`md_t_from` enum('gps','server') NOT NULL DEFAULT 'gps',
`md_valid_fix` tinyint(1) NOT NULL,
`md_ignition` tinyint(1) DEFAULT NULL,
`md_ignition_on_t` timestamp NULL DEFAULT NULL,
`md_ignition_off_t` timestamp NULL DEFAULT NULL,
`md_stationary` tinyint(1) DEFAULT NULL,
`md_lights` tinyint(1) DEFAULT NULL,
`md_battery` tinyint(1) DEFAULT NULL,
`md_panic_active` tinyint(1) DEFAULT NULL,
`md_panic_ack` tinyint(1) DEFAULT NULL,
`md_panic_active_t` timestamp NULL DEFAULT NULL,
`md_panic_cancel_t` timestamp NULL DEFAULT NULL,
`md_panic_ack_t` timestamp NULL DEFAULT NULL,
`md_stored_frame` tinyint(1) DEFAULT NULL,
`md_heading_change` tinyint(1) DEFAULT NULL,
`md_power_save` tinyint(1) DEFAULT NULL,
`md_gps_ariel_fault` enum('good','short','open') DEFAULT NULL,
`md_reset` tinyint(1) DEFAULT NULL,
`md_batt_volts` decimal(4,1) DEFAULT NULL,
`md_batt_percent` decimal(4,1) DEFAULT NULL,
`md_upgrade` tinyint(1) DEFAULT NULL,
`md_engine_by_volts` tinyint(1) DEFAULT NULL,
`md_flag1` tinyint(3) unsigned DEFAULT NULL,
`md_flag2` tinyint(3) unsigned DEFAULT NULL,
`md_tl_rec` bigint(20) unsigned DEFAULT NULL,
`md_server_name` varchar(40) DEFAULT NULL,
`mg_t` timestamp NULL DEFAULT NULL,
`mg_lat` decimal(9,6) DEFAULT NULL,
`mg_lon` decimal(9,6) DEFAULT NULL,
`mg_speed_n_h` decimal(5,1) DEFAULT NULL,
`mg_speed_km_h` decimal(5,1) DEFAULT NULL,
`mg_speed_m_h` decimal(5,1) DEFAULT NULL,
`mg_heading` int(11) DEFAULT NULL,
`mg_tl_rec` bigint(20) unsigned DEFAULT NULL,
`mg_server_name` varchar(40) DEFAULT NULL,
`mz_t` timestamp NULL DEFAULT NULL,
`mz_town` varchar(40) DEFAULT NULL,
`mz_county` varchar(40) DEFAULT NULL,
`mz_country` varchar(3) DEFAULT NULL,
`mz_pcode` varchar(16) DEFAULT NULL,
`mz_gaz_rec` int(11) DEFAULT NULL,
`mc_t` timestamp NULL DEFAULT NULL,
`mc_service_distance` int(11) DEFAULT NULL,
`mc_odo` decimal(11,3) DEFAULT NULL,
`mc_fuel` decimal(10,2) DEFAULT NULL,
`mc_fuel_level` decimal(4,1) DEFAULT NULL,
`mc_temperature` decimal(5,1) DEFAULT NULL,
`mc_cat_level` decimal(4,1) DEFAULT NULL,
`mc_axle_1` decimal(6,1) DEFAULT NULL,
`mc_axle_2` decimal(6,1) DEFAULT NULL,
`mc_axle_3` decimal(6,1) DEFAULT NULL,
`mc_axle_4` decimal(6,1) DEFAULT NULL,
`mc_flag1` tinyint(3) unsigned DEFAULT NULL,
`mc_flag2` tinyint(3) unsigned DEFAULT NULL,
`mc_engine_available` tinyint(1) DEFAULT NULL,
`mc_ending_on` tinyint(1) DEFAULT NULL,
`mc_engine_on` tinyint(1) DEFAULT NULL,
`mc_pto` tinyint(1) DEFAULT NULL,
`mc_from_trip` tinyint(1) DEFAULT NULL,
`mc_tl_rec` bigint(20) unsigned DEFAULT NULL,
`mc_server_name` varchar(40) DEFAULT NULL,
`ms_t` timestamp NULL DEFAULT NULL,
`ms_flag1` tinyint(3) unsigned DEFAULT NULL,
`ms_flag2` tinyint(3) unsigned DEFAULT NULL,
`ms_flag3` tinyint(3) unsigned DEFAULT NULL,
`ms_speed` decimal(5,1) DEFAULT NULL,
`ms_card_1` varchar(16) DEFAULT NULL,
`ms_card_2` varchar(16) DEFAULT NULL,
`ms_card_1_null` enum('not_null','unsupported_device','no_k_line','card_not_inserted','driver_unknown') DEFAULT NULL,
`ms_card_2_null` enum('not_null','unsupported_device','no_k_line','card_not_inserted','driver_unknown') DEFAULT NULL,
`ms_mem_code_1` int(11) DEFAULT NULL,
`ms_mem_code_2` int(11) DEFAULT NULL,
`ms_overspeed` tinyint(1) DEFAULT NULL,
`ms_time_state_1` enum('normal','15m_before_430h','430h','15m_before_900h','900h','15m_before_xxxh','xxxh') DEFAULT NULL,
`ms_time_state_2` enum('normal','15m_before_430h','430h','15m_before_900h','900h','15m_before_xxxh','xxxh') DEFAULT NULL,
`ms_card_1_mode` enum('w','r','d','a') DEFAULT NULL,
`ms_card_2_mode` enum('w','r','d','a') DEFAULT NULL,
`ms_tl_rec` bigint(20) unsigned DEFAULT NULL,
`ms_server_name` varchar(40) DEFAULT NULL,
`mt_t` timestamp NULL DEFAULT NULL,
`mt_trip_id` int(11) DEFAULT NULL,
`mt_source` enum('sdi1','sdi2','fc01') DEFAULT NULL,
`mt_trip_start` timestamp NULL DEFAULT NULL,
`mt_trip_end` timestamp NULL DEFAULT NULL,
`mt_trip_duration` int(11) DEFAULT NULL,
`mt_ignition_on_duration` int(11) DEFAULT NULL,
`mt_engine_on_duration` int(11) DEFAULT NULL,
`mt_motion_duration` int(11) DEFAULT NULL,
`mt_total_trip_duration` int(11) DEFAULT NULL,
`mt_pto_active_nomotion` int(11) DEFAULT NULL,
`mt_pto_active_motion` int(11) DEFAULT NULL,
`mt_trip_start_odometer` decimal(11,3) DEFAULT NULL,
`mt_trip_end_odometer` decimal(11,3) DEFAULT NULL,
`mt_trip_total_odometer` decimal(11,3) DEFAULT NULL,
`mt_trip_start_fuel` decimal(10,1) DEFAULT NULL,
`mt_trip_start_fuel_c` decimal(10,1) DEFAULT NULL,
`mt_trip_end_fuel` decimal(10,1) DEFAULT NULL,
`mt_trip_end_fuel_c` decimal(10,1) DEFAULT NULL,
`mt_trip_total_fuel` decimal(10,1) DEFAULT NULL,
`mt_trip_total_fuel_c` decimal(10,1) DEFAULT NULL,
`mt_trip_idle_fuel` decimal(10,1) DEFAULT NULL,
`mt_trip_idle_fuel_c` decimal(10,1) DEFAULT NULL,
`mt_trip_fuel_l_100km` decimal(6,1) DEFAULT NULL,
`mt_trip_fuel_l_100km_c` decimal(6,1) DEFAULT NULL,
`mt_trip_fuel_l_hr` decimal(6,1) DEFAULT NULL,
`mt_trip_fuel_l_hr_c` decimal(6,1) DEFAULT NULL,
`mt_trip_driving_duration` int(11) DEFAULT NULL,
`mt_trip_working_duration` int(11) DEFAULT NULL,
`mt_trip_rest_duration` int(11) DEFAULT NULL,
`mt_trip_available_duration` int(11) DEFAULT NULL,
`mt_trip_absolute_km_h` decimal(5,2) DEFAULT NULL,
`mt_trip_engine_on_km_h` decimal(5,2) DEFAULT NULL,
`mt_trip_motion_km_h` decimal(5,2) DEFAULT NULL,
`mt_trip_speed_zone1_cc_off_duration` int(11) DEFAULT NULL,
`mt_trip_speed_zone2_cc_off_duration` int(11) DEFAULT NULL,
`mt_trip_speed_zone3_cc_off_duration` int(11) DEFAULT NULL,
`mt_trip_speed_zone4_cc_off_duration` int(11) DEFAULT NULL,
`mt_trip_speed_total_cc_off_duration` int(11) DEFAULT NULL,
`mt_trip_speed_zone1_cc_on_duration` int(11) DEFAULT NULL,
`mt_trip_speed_zone2_cc_on_duration` int(11) DEFAULT NULL,
`mt_trip_speed_zone3_cc_on_duration` int(11) DEFAULT NULL,
`mt_trip_speed_zone4_cc_on_duration` int(11) DEFAULT NULL,
`mt_trip_speed_total_cc_on_duration` int(11) DEFAULT NULL,
`mt_trip_brake_pedal_count` int(11) DEFAULT NULL,
`mt_trip_brake_pedal_duration` int(11) DEFAULT NULL,
`mt_trip_accel_duration` int(11) DEFAULT NULL,
`mt_trip_decel_duration` int(11) DEFAULT NULL,
`mt_trip_accel_max_ms2` decimal(5,2) DEFAULT NULL,
`mt_trip_decel_max_ms2` decimal(5,2) DEFAULT NULL,
`mt_trip_acc_zone1_duration` int(11) DEFAULT NULL,
`mt_trip_acc_zone2_duration` int(11) DEFAULT NULL,
`mt_trip_acc_zone3_duration` int(11) DEFAULT NULL,
`mt_trip_acc_zone4_duration` int(11) DEFAULT NULL,
`mt_trip_acc_zone5_duration` int(11) DEFAULT NULL,
`mt_trip_acc_zone6_duration` int(11) DEFAULT NULL,
`mt_trip_acc_zone7_duration` int(11) DEFAULT NULL,
`mt_trip_throttle_zone1_duration` int(11) DEFAULT NULL,
`mt_trip_throttle_zone2_duration` int(11) DEFAULT NULL,
`mt_trip_throttle_zone3_duration` int(11) DEFAULT NULL,
`mt_trip_throttle_zone4_duration` int(11) DEFAULT NULL,
`mt_trip_engine_zone1_duration` int(11) DEFAULT NULL,
`mt_trip_engine_zone2_duration` int(11) DEFAULT NULL,
`mt_trip_engine_zone3_duration` int(11) DEFAULT NULL,
`mt_message_type` enum('T','U') DEFAULT NULL,
`mt_time_engine_torque_high` int(11) DEFAULT NULL,
`mt_time_in_high_gear` int(11) DEFAULT NULL,
`mt_time_high_engine_load` int(11) DEFAULT NULL,
`mt_time_low_load_high_throttle` int(11) DEFAULT NULL,
`mt_time_kick_down` int(11) DEFAULT NULL,
`mt_mean_engine_load` decimal(4,1) DEFAULT NULL,
`mt_server_name` varchar(40) DEFAULT NULL,
`created` timestamp NOT NULL DEFAULT '2000-01-01 00:00:00',
`updated` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
`ignore_pto` tinyint(1) NOT NULL DEFAULT '0',
`cfg_t` timestamp NULL DEFAULT NULL,
`cfg_apn` varchar(20) DEFAULT NULL,
`cfg_user` varchar(20) DEFAULT NULL,
`cfg_password` varchar(20) DEFAULT NULL,
`cfg_ignition_off_every` varchar(20) DEFAULT NULL,
`cfg_hd_change_rpt` varchar(20) DEFAULT NULL,
`cfg_can_fitted` varchar(20) DEFAULT NULL,
`cfg_extended_can` varchar(20) DEFAULT NULL,
`cfg_trip_every` varchar(20) DEFAULT NULL,
`cfg_tacho_download_enabled` varchar(20) DEFAULT NULL,
`cfg_can_comms_protocol` varchar(20) DEFAULT NULL,
`cfg_network_sel` varchar(20) DEFAULT NULL,
`cfg_ign_rep` varchar(20) DEFAULT NULL,
`cfg_debug` varchar(20) DEFAULT NULL,
`cfg_every` varchar(20) DEFAULT NULL,
`cfg_on_battery_backup_every` varchar(20) DEFAULT NULL,
`cfg_ign_sen` varchar(20) DEFAULT NULL,
`cfg_auto_reset` varchar(20) DEFAULT NULL,
`cfg_red_pwr_mod` varchar(20) DEFAULT NULL,
`cfg_red_pwr_min` varchar(20) DEFAULT NULL,
`cfg_eng_sense_enabled` varchar(20) DEFAULT NULL,
`cfg_eng_on_offset` varchar(20) DEFAULT NULL,
`cfg_panic_enabled` varchar(20) DEFAULT NULL,
`cfg_logging` varchar(20) DEFAULT NULL,
`cfg_rmt_logging` varchar(20) DEFAULT NULL,
`cfg_rmt_log_period` varchar(20) DEFAULT NULL,
`cfg_reset_fuel` varchar(20) DEFAULT NULL,
`cfg_alarm_time` varchar(20) DEFAULT NULL,
`cnf_auto_request` enum('no','yes_new','yes_done') NOT NULL DEFAULT 'no',
`cnf_auto_request_t` timestamp NULL DEFAULT NULL,
PRIMARY KEY (`terminal_id`),
KEY `comp_code` (`comp_code`,`veh_code`),
KEY `veh_rtid` (`veh_rtid`),
KEY `tag_master_terminal_id` (`tag_master_terminal_id`),
KEY `tag_master_veh_rtid` (`tag_master_veh_rtid`),
KEY `cnf_auto_request` (`cnf_auto_request`,`terminal_type_sub`)
) ENGINE=ndbcluster DEFAULT CHARSET=latin1



-----------------------------------------------------

Information Schema:

TABLE_CATALOG: def
TABLE_SCHEMA: ndb
TABLE_NAME: fl_state
TABLE_TYPE: BASE TABLE
ENGINE: ndbcluster
VERSION: 10
ROW_FORMAT: Dynamic
TABLE_ROWS: 6567
AVG_ROW_LENGTH: 844
DATA_LENGTH: 7405568
MAX_DATA_LENGTH: 0
INDEX_LENGTH: 0
DATA_FREE: 0
AUTO_INCREMENT: NULL
CREATE_TIME: NULL
UPDATE_TIME: NULL
CHECK_TIME: NULL
TABLE_COLLATION: latin1_swedish_ci
CHECKSUM: NULL
CREATE_OPTIONS:
TABLE_COMMENT:

I'm trying to configure my first MySQL cluster using the documentation at dev.mysql.com. I have my management and data nodes up, but I'm running into this error when I start the server on my SQL node:

[Note] NDB: NodeID is 4, management server '172.16.186.137:1186'
[Note] NDB[0]: NodeID: 4, no storage nodes connected (timed out)
[Warning] NDB: server id set to zero - changes logged to bin log with server id zero will be logged with another server id by slave mysqlds

I've found previous posts from people having this error that were behind a firewall, but my connectivity to the management host is good and its logs show the SQL node connecting:

[MgmtSrvr] INFO -- Nodeid 4 allocated for API at 172.16.186.136
[MgmtSrvr] INFO -- Node 4: mysqld --server-id=0
[MgmtSrvr] WARNING -- Node 2: Releasing node id allocation for node 4

## ndb_mgm output:

[ndbd(NDB)] 2 node(s)
id=2 @172.16.186.134 (mysql-5.5.30 ndb-7.2.12, Nodegroup: 0, Master)
id=3 @172.16.186.135 (mysql-5.5.30 ndb-7.2.12, Nodegroup: 0)

[ndb_mgmd(MGM)] 1 node(s)
id=1 @172.16.186.137 (mysql-5.5.30 ndb-7.2.12)

[mysqld(API)] 1 node(s)
id=4 (not connected, accepting connect from 172.16.186.136)

## The SQL node's my.cnf:

[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
user=mysql
symbolic-links=0
ndbcluster

[mysql_cluster]
ndb-connectstring=172.16.186.137

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid


I've exhausted myself going over the confs and everything appears to be set correctly. If anyone could offer some suggestions, it would be greatly appreciated.

Dear Cluster,

I am trying to understand what index types are available on Cluster for the user to make use of. But I have not having much luck finding accurate and recent documentation. I wonder whether any members know?

I believe that Cluster used to use a hash-index for both Primary and Secondary indexes?

But now is it possible to use a T-Tree index on Primary and/or Secondary?

If this is the case, what is the syntax? I have trying "USING {BTREE | HASH}" on a "CREATE TABLE" without conclusive results.

Of if anybody has a good URL on the subject, that would be very useful.

Many thanks!!

Ben.

Hello, I'm new of cluster.
I have a cluster of two servers.
Server 1: management, ndb and mysql
Server 2: ndb e mysql
The cluster works good after same change to default value.
A customer has a website on the two servers.
Sometimes, after/during big queries, nbd comes up tp 120% of the cpu and everithing is slow.
If i kill Server2 everything works better, but sometime the same thing happen also on Server1.
I cannot understand why. ANy ideas?
Thanks

Recently we found that one of the cluster node disconnect at specific time in midnight. Checking the management node log, following error message was found:
2013-04-14 01:03:16 [MgmtSrvr] ALERT -- Node 3: Forced node shutdown complete
d. Caused by error 2303: 'System error, node killed during node restart by other
node(Internal error, programming error or missing error message, please report
a bug). Temporary error, restart node'.
2013-04-16 01:03:09 [MgmtSrvr] ALERT -- Node 3: Forced node shutdown complete
d. Caused by error 2303: 'System error, node killed during node restart by other
node(Internal error, programming error or missing error message, please report
a bug). Temporary error, restart node'.
2013-04-17 01:02:30 [MgmtSrvr] ALERT -- Node 3: Forced node shutdown complete
d. Caused by error 2303: 'System error, node killed during node restart by other
node(Internal error, programming error or missing error message, please report
a bug). Temporary error, restart node'.
-------------------------------------------------------------------------

The log pattern before error happen:
2013-04-17 00:58:45 [MgmtSrvr] INFO -- Node 2: Communication to Node 5 closed
2013-04-17 00:58:45 [MgmtSrvr] INFO -- Node 3: Communication to Node 5 closed
2013-04-17 00:58:45 [MgmtSrvr] ALERT -- Node 2: Node 5 Disconnected
2013-04-17 00:58:46 [MgmtSrvr] ALERT -- Node 3: Node 5 Disconnected
2013-04-17 00:58:46 [MgmtSrvr] INFO -- Node 3: Communication to Node 5 opened
2013-04-17 00:58:46 [MgmtSrvr] INFO -- Node 3: Node 5 Connected
2013-04-17 00:58:47 [MgmtSrvr] INFO -- Node 3: Node 5: API mysql-5.1.56 ndb-7.1.15
2013-04-17 00:58:48 [MgmtSrvr] INFO -- Node 2: Communication to Node 5 opened
2013-04-17 00:58:48 [MgmtSrvr] INFO -- Node 2: Node 5 Connected
2013-04-17 00:58:48 [MgmtSrvr] INFO -- Node 2: Node 5: API mysql-5.1.56 ndb-7.1.15
2013-04-17 00:58:57 [MgmtSrvr] WARNING -- Node 2: Node 4 missed heartbeat 2
2013-04-17 00:58:58 [MgmtSrvr] WARNING -- Node 2: Node 5 missed heartbeat 2
2013-04-17 00:59:01 [MgmtSrvr] WARNING -- Node 2: Node 5 missed heartbeat 2
2013-04-17 00:59:20 [MgmtSrvr] WARNING -- Node 2: Node 4 missed heartbeat 2
2013-04-17 00:59:22 [MgmtSrvr] WARNING -- Node 2: Node 5 missed heartbeat 2
2013-04-17 00:59:22 [MgmtSrvr] WARNING -- Node 2: Node 5 missed heartbeat 3
2013-04-17 00:59:24 [MgmtSrvr] WARNING -- Node 2: Node 5 missed heartbeat 2
2013-04-17 00:59:49 [MgmtSrvr] WARNING -- Node 2: Node 4 missed heartbeat 2
2013-04-17 00:59:56 [MgmtSrvr] WARNING -- Node 2: Node 4 missed heartbeat 2
2013-04-17 00:59:56 [MgmtSrvr] WARNING -- Node 2: Node 5 missed heartbeat 2
2013-04-17 01:00:05 [MgmtSrvr] WARNING -- Node 3: GCP Monitor: GCP_COMMIT lag 1 seconds (max lag: 27)
2013-04-17 01:00:13 [MgmtSrvr] INFO -- Node 3: Local checkpoint 8535 completed
2013-04-17 01:00:16 [MgmtSrvr] INFO -- Node 3: Local checkpoint 8536 started. Keep GCI = 396148 oldest restorable GCI = 396103
2013-04-17 01:00:28 [MgmtSrvr] WARNING -- Node 3: GCP Monitor: GCP_COMMIT lag 1 seconds (max lag: 27)
2013-04-17 01:00:35 [MgmtSrvr] WARNING -- Node 2: Node 5 missed heartbeat 2
2013-04-17 01:00:36 [MgmtSrvr] WARNING -- Node 3: GCP Monitor: GCP_COMMIT lag 0 seconds (max lag: 27)
2013-04-17 01:00:39 [MgmtSrvr] WARNING -- Node 3: GCP Monitor: GCP_COMMIT lag 0 seconds (max lag: 27)
2013-04-17 01:00:41 [MgmtSrvr] WARNING -- Node 3: GCP Monitor: GCP_COMMIT lag 0 seconds (max lag: 27)
2013-04-17 01:00:44 [MgmtSrvr] WARNING -- Node 3: GCP Monitor: GCP_COMMIT lag 1 seconds (max lag: 27)
2013-04-17 01:00:49 [MgmtSrvr] WARNING -- Node 3: GCP Monitor: GCP_COMMIT lag 1 seconds (max lag: 27)
2013-04-17 01:00:54 [MgmtSrvr] WARNING -- Node 3: GCP Monitor: GCP_COMMIT lag 0 seconds (max lag: 27)
2013-04-17 01:00:59 [MgmtSrvr] WARNING -- Node 3: GCP Monitor: GCP_COMMIT lag 0 seconds (max lag: 27)
2013-04-17 01:01:00 [MgmtSrvr] WARNING -- Node 2: Node 5 missed heartbeat 2
2013-04-17 01:01:00 [MgmtSrvr] WARNING -- Node 2: Node 5 missed heartbeat 3
2013-04-17 01:01:01 [MgmtSrvr] WARNING -- Node 2: Node 4 missed heartbeat 2
2013-04-17 01:01:01 [MgmtSrvr] WARNING -- Node 2: Node 5 missed heartbeat 4
2013-04-17 01:01:01 [MgmtSrvr] ALERT -- Node 2: Node 5 declared dead due to missed heartbeat
2013-04-17 01:01:01 [MgmtSrvr] INFO -- Node 2: Communication to Node 5 closed
2013-04-17 01:01:01 [MgmtSrvr] INFO -- Node 3: Communication to Node 5 closed
2013-04-17 01:01:01 [MgmtSrvr] ALERT -- Node 3: Node 5 Disconnected
2013-04-17 01:01:01 [MgmtSrvr] ALERT -- Node 2: Node 5 Disconnected
2013-04-17 01:01:01 [MgmtSrvr] WARNING -- Node 2: Node 4 missed heartbeat 3
2013-04-17 01:01:02 [MgmtSrvr] INFO -- Node 2: Communication to Node 5 opened
2013-04-17 01:01:02 [MgmtSrvr] INFO -- Node 2: Node 5 Connected
2013-04-17 01:01:02 [MgmtSrvr] INFO -- Node 2: Node 5: API mysql-5.1.56 ndb-7.1.15
2013-04-17 01:01:03 [MgmtSrvr] INFO -- Node 3: Communication to Node 5 opened
2013-04-17 01:01:00 [MgmtSrvr] INFO -- Node 3: Node 5 Connected
2013-04-17 01:01:00 [MgmtSrvr] INFO -- Node 3: Node 5: API mysql-5.1.56 ndb-7.1.15
2013-04-17 01:01:00 [MgmtSrvr] WARNING -- Node 3: GCP Monitor: GCP_COMMIT lag 1 seconds (max lag: 27)
2013-04-17 01:01:04 [MgmtSrvr] WARNING -- Node 2: Node 5 missed heartbeat 2
2013-04-17 01:01:05 [MgmtSrvr] WARNING -- Node 3: GCP Monitor: GCP_COMMIT lag 1 seconds (max lag: 27)
2013-04-17 01:01:08 [MgmtSrvr] WARNING -- Node 2: Node 5 missed heartbeat 2
2013-04-17 01:01:08 [MgmtSrvr] WARNING -- Node 3: GCP Monitor: GCP_COMMIT lag 0 seconds (max lag: 27)
2013-04-17 01:01:09 [MgmtSrvr] WARNING -- Node 2: Node 5 missed heartbeat 2
2013-04-17 01:01:12 [MgmtSrvr] WARNING -- Node 2: Node 5 missed heartbeat 2
2013-04-17 01:01:14 [MgmtSrvr] WARNING -- Node 3: GCP Monitor: GCP_SAVE lag 60 seconds (max lag: 145s)
2013-04-17 01:01:14 [MgmtSrvr] WARNING -- Node 3: GCP Monitor: GCP_COMMIT lag 0 seconds (max lag: 27)
2013-04-17 01:01:17 [MgmtSrvr] WARNING -- Node 3: GCP Monitor: GCP_COMMIT lag 1 seconds (max lag: 27)
2013-04-17 01:01:20 [MgmtSrvr] WARNING -- Node 3: GCP Monitor: GCP_COMMIT lag 1 seconds (max lag: 27)
2013-04-17 01:01:23 [MgmtSrvr] WARNING -- Node 3: GCP Monitor: GCP_COMMIT lag 0 seconds (max lag: 27)
2013-04-17 01:01:26 [MgmtSrvr] WARNING -- Node 3: GCP Monitor: GCP_COMMIT lag 0 seconds (max lag: 27)
2013-04-17 01:01:30 [MgmtSrvr] WARNING -- Node 3: GCP Monitor: GCP_COMMIT lag 0 seconds (max lag: 27)
2013-04-17 01:01:33 [MgmtSrvr] WARNING -- Node 3: GCP Monitor: GCP_SAVE lag 120 seconds (max lag: 145s)
2013-04-17 01:01:33 [MgmtSrvr] WARNING -- Node 3: GCP Monitor: GCP_COMMIT lag 0 seconds (max lag: 27)
2013-04-17 01:01:35 [MgmtSrvr] WARNING -- Node 3: GCP Monitor: GCP_COMMIT lag 0 seconds (max lag: 27)
2013-04-17 01:01:39 [MgmtSrvr] WARNING -- Node 2: Node 4 missed heartbeat 2
2013-04-17 01:01:40 [MgmtSrvr] WARNING -- Node 2: Node 5 missed heartbeat 2
2013-04-17 01:01:43 [MgmtSrvr] WARNING -- Node 3: GCP Monitor: GCP_COMMIT lag 0 seconds (max lag: 27)
2013-04-17 01:01:43 [MgmtSrvr] WARNING -- Node 2: Node 5 missed heartbeat 2
2013-04-17 01:01:43 [MgmtSrvr] WARNING -- Node 2: Node 5 missed heartbeat 3
2013-04-17 01:01:44 [MgmtSrvr] WARNING -- Node 2: Node 4 missed heartbeat 2
2013-04-17 01:01:53 [MgmtSrvr] ALERT -- Node 1: Node 3 Disconnected
2013-04-17 01:01:56 [MgmtSrvr] WARNING -- Node 2: Node 3 missed heartbeat 2
2013-04-17 01:02:00 [MgmtSrvr] WARNING -- Node 2: Node 3 missed heartbeat 3
2013-04-17 01:02:04 [MgmtSrvr] WARNING -- Node 2: Node 3 missed heartbeat 4
2013-04-17 01:02:04 [MgmtSrvr] ALERT -- Node 2: Node 3 declared dead due to missed heartbeat
2013-04-17 01:02:04 [MgmtSrvr] INFO -- Node 2: Communication to Node 3 closed
2013-04-17 01:02:04 [MgmtSrvr] ALERT -- Node 2: Network partitioning - arbitration required
2013-04-17 01:02:04 [MgmtSrvr] INFO -- Node 2: President restarts arbitration thread [state=7]
2013-04-17 01:02:04 [MgmtSrvr] ALERT -- Node 2: Arbitration won - positive reply from node 1
2013-04-17 01:02:04 [MgmtSrvr] INFO -- Node 2: GCP Take over started
2013-04-17 01:02:04 [MgmtSrvr] INFO -- Node 2: Node 2 taking over as DICT master
2013-04-17 01:02:04 [MgmtSrvr] INFO -- Node 2: GCP Monitor: Computed max GCP_SAVE lag to 145 seconds
2013-04-17 01:02:04 [MgmtSrvr] INFO -- Node 2: GCP Monitor: Computed max GCP_COMMIT lag to 27 seconds
2013-04-17 01:02:04 [MgmtSrvr] INFO -- Node 2: GCP Take over completed
2013-04-17 01:02:04 [MgmtSrvr] INFO -- Node 2: kk: 396202/112 1 0
2013-04-17 01:02:04 [MgmtSrvr] ALERT -- Node 2: Node 3 Disconnected
2013-04-17 01:02:04 [MgmtSrvr] INFO -- Node 2: Started arbitrator node 1 [ticket=54c500233951f1d4]
2013-04-17 01:02:30 [MgmtSrvr] ALERT -- Node 3: Forced node shutdown completed. Caused by error 2303: 'System error, node killed during node restart by other node(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'.
2013-04-17 01:02:31 [MgmtSrvr] INFO -- Mgmt server state: nodeid 3 freed, m_reserved_nodes 1 and 2.
2013-04-17 01:02:32 [MgmtSrvr] WARNING -- Node 2: Failure handling of node 3 has not completed in 1 min - state = 6
2013-04-17 01:02:34 [MgmtSrvr] INFO -- Node 2: LCP Take over started
------------------------------------------------------------------------

ndb_3_out.log
------------------------------------------------------------------------
2013-04-17 01:01:59 [ndbd] WARNING -- Time moved forward with 1533 ms
2013-04-17 01:01:59 [ndbd] WARNING -- timerHandlingLab now: 961668499 sent: 961666952 diff: 1547
2013-04-17 01:01:59 [ndbd] WARNING -- timerHandlingLab now: 961667218 sent: 961668711 diff: -1493
2013-04-17 01:01:59 [ndbd] WARNING -- Time moved backwards with 1503 ms
2013-04-17 01:01:59 [ndbd] WARNING -- Time moved forward with 1520 ms
2013-04-17 01:01:59 [ndbd] WARNING -- timerHandlingLab now: 961669290 sent: 961667770 diff: 1520
2013-04-17 01:02:00 [ndbd] WARNING -- timerHandlingLab now: 961667832 sent: 961669331 diff: -1499
2013-04-17 01:02:00 [ndbd] WARNING -- Time moved backwards with 1509 ms
2013-04-17 01:02:01 [ndbd] WARNING -- Time moved forward with 1526 ms
2013-04-17 01:02:01 [ndbd] WARNING -- timerHandlingLab now: 961670318 sent: 961668783 diff: 1535
2013-04-17 01:02:01 [ndbd] WARNING -- Time moved backwards with 1502 ms
2013-04-17 01:02:01 [ndbd] WARNING -- Time moved forward with 1523 ms
2013-04-17 01:02:01 [ndbd] WARNING -- Time moved backwards with 1506 ms
2013-04-17 01:01:56 [ndbd] WARNING -- timerHandlingLab now: 961668994 sent: 961670477 diff: -1483
2013-04-17 01:01:58 [ndbd] WARNING -- Time moved forward with 1530 ms
2013-04-17 01:01:58 [ndbd] WARNING -- timerHandlingLab now: 961672212 sent: 961670671 diff: 1541
2013-04-17 01:01:58 [ndbd] WARNING -- Time moved backwards with 1512 ms
2013-04-17 01:01:58 [ndbd] WARNING -- timerHandlingLab now: 961670724 sent: 961672212 diff: -1488
2013-04-17 01:01:59 [ndbd] WARNING -- Time moved forward with 1522 ms
2013-04-17 01:01:59 [ndbd] WARNING -- timerHandlingLab now: 961672969 sent: 961671447 diff: 1522
2013-04-17 01:01:59 [ndbd] INFO -- findNeighbours from: 4891 old (left: 3 right: 3) new (65535 65535)
2013-04-17 01:01:59 [ndbd] INFO -- Watchdog: User time: 15145 System time: 34380
2013-04-17 01:01:59 [ndbd] WARNING -- Watchdog: Warning overslept 1614 ms, expected 100 ms.
start_resend(1, empty bucket (396202/112 396202/111) -> active
REMOVING lcp: 8536 from table: 2 frag: 0 node: 3
REMOVING lcp: 8536 from table: 2 frag: 1 node: 3
Finished with handling node-failure
execGCP_NOMORETRANS(396202/112) c_ongoing_take_over_cnt -> seize
2013-04-17 01:01:59 [ndbd] WARNING -- Time moved backwards with 1462 ms
2013-04-17 01:01:59 [ndbd] WARNING -- timerHandlingLab now: 961671549 sent: 961672998 diff: -1449
completing gcp 396202/112 in execTAKE_OVERTCCONF
REMOVING lcp: 8536 from table: 3 frag: 0 node: 3
REMOVING lcp: 8536 from table: 3 frag: 1 node: 3
REMOVING lcp: 8536 from table: 4 frag: 0 node: 3
REMOVING lcp: 8536 from table: 4 frag: 1 node: 3
REMOVING lcp: 8536 from table: 5 frag: 0 node: 3
REMOVING lcp: 8536 from table: 5 frag: 1 node: 3
REMOVING lcp: 8536 from table: 6 frag: 0 node: 3
REMOVING lcp: 8536 from table: 6 frag: 1 node: 3
REMOVING lcp: 8536 from table: 7 frag: 0 node: 3
REMOVING lcp: 8536 from table: 7 frag: 1 node: 3
REMOVING lcp: 8536 from table: 9 frag: 0 node: 3
REMOVING lcp: 8536 from table: 9 frag: 1 node: 3
2013-04-17 01:01:59 [ndbd] WARNING -- Time moved forward with 1537 ms
2013-04-17 01:01:59 [ndbd] WARNING -- timerHandlingLab now: 961673333 sent: 961671787 diff: 1546
2013-04-17 01:01:59 [ndbd] WARNING -- Time moved backwards with 1511 ms
------------------------------------------------------------------------
ndb_3_out.log
------------------------------------------------------------------------
2013-04-17 01:01:30 [ndbd] WARNING -- timerHandlingLab now: 969629986 sent: 969632245 diff: -2259
2013-04-17 01:01:30 [ndbd] WARNING -- Time moved forward with 2286 ms
2013-04-17 01:01:30 [ndbd] INFO -- Watchdog: User time: 162515 System time: 425964
2013-04-17 01:01:30 [ndbd] WARNING -- Watchdog: Warning overslept 2376 ms, expected 100 ms.
2013-04-17 01:01:30 [ndbd] WARNING -- timerHandlingLab now: 969632301 sent: 969630006 diff: 2295
2013-04-17 01:01:30 [ndbd] WARNING -- Time moved backwards with 2272 ms
2013-04-17 01:01:30 [ndbd] WARNING -- timerHandlingLab now: 969630050 sent: 969632302 diff: -2252
2013-04-17 01:01:30 [ndbd] WARNING -- Time moved forward with 2283 ms
2013-04-17 01:01:30 [ndbd] WARNING -- timerHandlingLab now: 969632557 sent: 969630263 diff: 2294
2013-04-17 01:01:30 [ndbd] WARNING -- Time moved backwards with 2272 ms
2013-04-17 01:01:30 [ndbd] WARNING -- timerHandlingLab now: 969630302 sent: 969632558 diff: -2256
2013-04-17 01:01:31 [ndbd] WARNING -- Time moved forward with 2286 ms
2013-04-17 01:01:31 [ndbd] WARNING -- timerHandlingLab now: 969633055 sent: 969630751 diff: 2304
2013-04-17 01:01:31 [ndbd] WARNING -- Time moved backwards with 2270 ms
2013-04-17 01:01:31 [ndbd] WARNING -- timerHandlingLab now: 969630828 sent: 969633078 diff: -2250
2013-04-17 01:01:31 [ndbd] WARNING -- Time moved forward with 2293 ms
2013-04-17 01:01:31 [ndbd] WARNING -- timerHandlingLab now: 969633549 sent: 969631234 diff: 2315
2013-04-17 01:01:31 [ndbd] WARNING -- Time moved backwards with 2272 ms
2013-04-17 01:01:31 [ndbd] WARNING -- timerHandlingLab now: 969631287 sent: 969633549 diff: -2262
2013-04-17 01:01:31 [ndbd] WARNING -- Time moved forward with 2282 ms
2013-04-17 01:01:31 [ndbd] WARNING -- timerHandlingLab now: 969633590 sent: 969631297 diff: 2293
2013-04-17 01:01:31 [ndbd] WARNING -- Time moved backwards with 2272 ms
2013-04-17 01:01:31 [ndbd] WARNING -- timerHandlingLab now: 969631406 sent: 969633658 diff: -2252
2013-04-17 01:01:33 [ndbd] WARNING -- Time moved forward with 2283 ms
2013-04-17 01:01:33 [ndbd] WARNING -- Time moved backwards with 1283 ms
2013-04-17 01:01:34 [ndbd] WARNING -- timerHandlingLab now: 969636046 sent: 969633744 diff: 2302
2013-04-17 01:01:34 [ndbd] WARNING -- Time moved forward with 2292 ms
2013-04-17 01:01:34 [ndbd] WARNING -- Time moved backwards with 2270 ms
2013-04-17 01:01:34 [ndbd] WARNING -- timerHandlingLab now: 969633790 sent: 969636046 diff: -2256
2013-04-17 01:01:38 [ndbd] WARNING -- Time moved forward with 2284 ms
2013-04-17 01:01:38 [ndbd] WARNING -- timerHandlingLab now: 969639889 sent: 969637605 diff: 2284
2013-04-17 01:01:38 [ndbd] WARNING -- Time moved backwards with 2270 ms
2013-04-17 01:01:38 [ndbd] WARNING -- timerHandlingLab now: 969637643 sent: 969639890 diff: -2247
2013-04-17 01:01:38 [ndbd] WARNING -- Time moved forward with 2284 ms
2013-04-17 01:01:38 [ndbd] WARNING -- timerHandlingLab now: 969640512 sent: 969638217 diff: 2295
2013-04-17 01:01:38 [ndbd] WARNING -- Time moved backwards with 2272 ms
2013-04-17 01:01:38 [ndbd] WARNING -- timerHandlingLab now: 969638251 sent: 969640512 diff: -2261
2013-04-17 01:01:40 [ndbd] WARNING -- timerHandlingLab now: 969642658 sent: 969640363 diff: 2295
2013-04-17 01:01:41 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=99
2013-04-17 01:01:41 [ndbd] INFO -- Watchdog: User time: 162516 System time: 425966
2013-04-17 01:01:41 [ndbd] WARNING -- timerHandlingLab now: 969640826 sent: 969642658 diff: -1832
2013-04-17 01:01:41 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425969
2013-04-17 01:01:41 [ndbd] WARNING -- Watchdog: Warning overslept 427 ms, expected 100 ms.
2013-04-17 01:01:42 [ndbd] WARNING -- Time moved forward with 2284 ms
2013-04-17 01:01:42 [ndbd] WARNING -- timerHandlingLab now: 969644280 sent: 969641992 diff: 2288
2013-04-17 01:01:42 [ndbd] WARNING -- Time moved backwards with 2272 ms
2013-04-17 01:01:42 [ndbd] WARNING -- timerHandlingLab now: 969642029 sent: 969644281 diff: -2252
2013-04-17 01:01:43 [ndbd] WARNING -- Time moved forward with 2283 ms
2013-04-17 01:01:43 [ndbd] WARNING -- timerHandlingLab now: 969645600 sent: 969643311 diff: 2289
c_nodeStartMaster.blockGcp: 0 4294967040
m_gcp_save.m_counter: 1450 m_gcp_save.m_max_lag: 1450
m_micro_gcp.m_counter: 13 m_micro_gcp.m_max_lag: 271
m_gcp_save.m_state: 3
m_gcp_save.m_master.m_state: 3
m_micro_gcp.m_state: 1
m_micro_gcp.m_master.m_state: 1
c_COPY_GCIREQ_Counter = [SignalCounter: m_count=1 0000000000000008]
c_COPY_TABREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_CREATE_FRAGREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_DIH_SWITCH_REPLICA_REQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_EMPTY_LCP_REQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_GCP_COMMIT_Counter = [SignalCounter: m_count=0 0000000000000000]
c_GCP_PREPARE_Counter = [SignalCounter: m_count=1 0000000000000004]
c_GCP_SAVEREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_SUB_GCP_COMPLETE_REP_Counter = [SignalCounter: m_count=0 0000000000000000]
c_INCL_NODEREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_MASTER_GCPREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_MASTER_LCPREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_START_INFOREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_START_RECREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_STOP_ME_REQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_TC_CLOPSIZEREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_TCGETOPSIZEREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
m_copyReason: 3 m_waiting: 0 0
c_copyGCISlave: sender{Data, Ref} 3 f60003 reason: 3 nextWord: 0
Detected GCP stop(3)...sending kill to [SignalCounter: m_count=1 0000000000000008]
c_nodeStartMaster.blockGcp: 0 4294967040
m_gcp_save.m_counter: 0 m_gcp_save.m_max_lag: 1450
m_micro_gcp.m_counter: 0 m_micro_gcp.m_max_lag: 271
m_gcp_save.m_state: 3
m_gcp_save.m_master.m_state: 3
m_micro_gcp.m_state: 1
m_micro_gcp.m_master.m_state: 1
c_COPY_GCIREQ_Counter = [SignalCounter: m_count=1 0000000000000008]
c_COPY_TABREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_CREATE_FRAGREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_DIH_SWITCH_REPLICA_REQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_EMPTY_LCP_REQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_GCP_COMMIT_Counter = [SignalCounter: m_count=0 0000000000000000]
c_GCP_PREPARE_Counter = [SignalCounter: m_count=1 0000000000000004]
c_GCP_SAVEREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_SUB_GCP_COMPLETE_REP_Counter = [SignalCounter: m_count=0 0000000000000000]
c_INCL_NODEREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_MASTER_GCPREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_MASTER_LCPREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_START_INFOREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_START_RECREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_STOP_ME_REQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_TC_CLOPSIZEREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_TCGETOPSIZEREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
m_copyReason: 3 m_waiting: 0 0
c_copyGCISlave: sender{Data, Ref} 3 f60003 reason: 3 nextWord: 0
file[0] status: 2 type: 1 reqStatus: 0 file1: 2 1 4
c_nodeStartMaster.blockGcp: 0 4294967040
m_gcp_save.m_counter: 0 m_gcp_save.m_max_lag: 1450
m_micro_gcp.m_counter: 0 m_micro_gcp.m_max_lag: 271
m_gcp_save.m_state: 3
m_gcp_save.m_master.m_state: 3
m_micro_gcp.m_state: 1
m_micro_gcp.m_master.m_state: 1
c_COPY_GCIREQ_Counter = [SignalCounter: m_count=1 0000000000000008]
c_COPY_TABREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_CREATE_FRAGREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_DIH_SWITCH_REPLICA_REQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_EMPTY_LCP_REQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_GCP_COMMIT_Counter = [SignalCounter: m_count=0 0000000000000000]
c_GCP_PREPARE_Counter = [SignalCounter: m_count=1 0000000000000004]
c_GCP_SAVEREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_SUB_GCP_COMPLETE_REP_Counter = [SignalCounter: m_count=0 0000000000000000]
c_INCL_NODEREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_MASTER_GCPREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_MASTER_LCPREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_START_INFOREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_START_RECREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_STOP_ME_REQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_TC_CLOPSIZEREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_TCGETOPSIZEREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
m_copyReason: 3 m_waiting: 0 0
c_copyGCISlave: sender{Data, Ref} 3 f60003 reason: 3 nextWord: 0
2013-04-17 01:01:44 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=100
2013-04-17 01:01:44 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:44 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=200
2013-04-17 01:01:44 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:44 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=299
2013-04-17 01:01:44 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:44 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=401
2013-04-17 01:01:44 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:44 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=501
2013-04-17 01:01:44 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:44 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=601
2013-04-17 01:01:44 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:44 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=701
2013-04-17 01:01:44 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:44 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=802
2013-04-17 01:01:44 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:44 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=903
2013-04-17 01:01:44 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:44 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=1002
2013-04-17 01:01:44 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:45 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=1103
2013-04-17 01:01:45 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:45 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=1208
2013-04-17 01:01:45 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:45 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=1319
2013-04-17 01:01:45 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:45 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=1420
2013-04-17 01:01:45 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:45 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=1520
2013-04-17 01:01:45 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:45 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=1619
2013-04-17 01:01:45 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:45 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=1719
2013-04-17 01:01:45 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:45 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=1820
2013-04-17 01:01:45 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:45 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=1920
2013-04-17 01:01:45 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:46 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=2021
2013-04-17 01:01:46 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:46 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=2121
2013-04-17 01:01:46 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:46 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=2220
2013-04-17 01:01:46 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:46 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=2320
2013-04-17 01:01:46 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:46 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=2420
2013-04-17 01:01:46 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:46 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=2520
2013-04-17 01:01:46 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:46 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=2621
2013-04-17 01:01:46 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:46 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=2720
2013-04-17 01:01:46 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:46 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=2820
2013-04-17 01:01:46 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:46 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=2920
2013-04-17 01:01:46 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:47 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=3020
2013-04-17 01:01:47 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:47 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=3121
2013-04-17 01:01:47 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:47 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=3220
2013-04-17 01:01:47 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:47 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=3321
2013-04-17 01:01:47 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:47 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=3422
2013-04-17 01:01:47 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:47 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=3521
2013-04-17 01:01:47 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:47 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=3621
2013-04-17 01:01:47 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:47 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=3722
2013-04-17 01:01:47 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:47 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=3822
2013-04-17 01:01:47 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:47 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=3922
2013-04-17 01:01:47 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:48 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=4022
2013-04-17 01:01:48 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:48 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=4122
2013-04-17 01:01:48 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:48 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=4222
2013-04-17 01:01:48 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:48 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=4322
2013-04-17 01:01:48 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:48 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=4422
2013-04-17 01:01:48 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:48 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=4521
2013-04-17 01:01:48 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:48 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=4621
2013-04-17 01:01:48 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:48 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=4722
2013-04-17 01:01:48 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:48 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=4823
2013-04-17 01:01:48 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:48 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=4923
2013-04-17 01:01:48 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:49 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=5023
2013-04-17 01:01:49 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:49 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=5123
2013-04-17 01:01:49 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:49 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=5223
2013-04-17 01:01:49 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:49 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=5323
2013-04-17 01:01:49 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425972
2013-04-17 01:01:49 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=5423
2013-04-17 01:01:49 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425972
2013-04-17 01:01:49 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=5524
2013-04-17 01:01:49 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425972
2013-04-17 01:01:49 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=5623
2013-04-17 01:01:49 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425972
2013-04-17 01:01:49 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=5724
2013-04-17 01:01:49 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425972
2013-04-17 01:01:49 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=5823
2013-04-17 01:01:49 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425972
2013-04-17 01:01:49 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=5924
2013-04-17 01:01:49 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425972
2013-04-17 01:01:50 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=6023
2013-04-17 01:01:50 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425972
2013-04-17 01:01:50 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Print Job Buffers at crash elapsed=100
2013-04-17 01:01:50 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425972
2013-04-17 01:01:50 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Print Job Buffers at crash elapsed=200
2013-04-17 01:01:50 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425972
2013-04-17 01:01:50 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Print Job Buffers at crash elapsed=299
2013-04-17 01:01:50 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425972
2013-04-17 01:01:50 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Print Job Buffers at crash elapsed=399
2013-04-17 01:01:50 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425972
2013-04-17 01:01:50 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Print Job Buffers at crash elapsed=499
2013-04-17 01:01:50 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425972
2013-04-17 01:01:50 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Print Job Buffers at crash elapsed=600
2013-04-17 01:01:50 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425972
2013-04-17 01:01:50 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Print Job Buffers at crash elapsed=700
2013-04-17 01:01:50 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425972
2013-04-17 01:01:50 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Print Job Buffers at crash elapsed=800
2013-04-17 01:01:50 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425972
2013-04-17 01:01:51 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Print Job Buffers at crash elapsed=899
2013-04-17 01:01:51 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425972
2013-04-17 01:01:51 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Print Job Buffers at crash elapsed=999
2013-04-17 01:01:51 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425972
2013-04-17 01:01:51 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Print Job Buffers at crash elapsed=1099
2013-04-17 01:01:51 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425972
2013-04-17 01:01:51 [ndbd] INFO -- Node 3 killed this node because GCP stop was detected
2013-04-17 01:01:51 [ndbd] INFO -- NDBCNTR (Line: 276) 0x00000002
2013-04-17 01:01:51 [ndbd] INFO -- Error handler shutting down system
2013-04-17 01:01:53 [ndbd] INFO -- Error handler shutdown completed - exiting
2013-04-17 01:02:29 [ndbd] ALERT -- Node 3: Forced node shutdown completed. Caused by error 2303: 'System error, node killed during node restart by other node(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'.
-------------------------------------------------------------------------

The following is the management node config.ini:
-------------------------------------------------------------------------
[ndbd default]
NoOfReplicas=2 # Number of replicas
DataMemory=2048M
IndexMemory=20M
RedoBuffer=512M
TimeBetweenLocalCheckpoints=0
NoOfFragmentLogFiles=64
HeartbeatIntervalDbDb=5000
# TCP/IP options:
[tcp default]

[ndb_mgmd]
hostname=10.0.104.132 # Hostname or IP address of management node
datadir=/var/lib/mysql-cluster # Directory for management node log files
# Options for data node "A":
[ndbd]
hostname=10.0.104.123 # Hostname or IP address
datadir=/usr/local/mysql/data # Directory for this data node's data files
[ndbd]
hostname=10.0.104.122 # Hostname or IP address
datadir=/usr/local/mysql/data # Directory for this data node's data files

[mysqld]
[mysqld]
-------------------------------------------------------------------------

Checked that there is no schedule job execute at that moment. The network status is good during incident period.

Is there any suggestion to troubleshoot the problem? Thanks

Overview:
MySQL Cluster 7.3 DMR2: Increasing Developer Flexibility and Simplicity

MySQL Cluster: Auto-Installer
http://www.clusterdb.com/mysql-cluster/mysql-cluster-7-3-auto-installer/

MySQL Cluster: Foreign Keys
http://www.clusterdb.com/mysql-cluster/foreign-keys-in-mysql-cluster/

MySQL Cluster: Node.js
https://blogs.oracle.com/MySQL/entry/mysql_cluster_tutorial_nosql_api

Hi,

I downloaded the latest milestone release this morning and extracted it to my machine. I notice that there isn't the ndb_setup python script in the bin directory.

Has the version not yet been updated for the Linux-Generic dist?

I have the 7.2.8 version from the labs build. Can I use that instead? Would I just need the python script, or are there a lot of other includes?

Cluster,

In 7.2.10, I have experienced a serious mysqld lockup. All queries where throwing:

Got error 4547 'RecordSpecification has overlapping offsets' from NDBCLUSTER

This occurred after I completed some DDL on another mysqld server connected to the same cluster:

ALTER TABLE fl_state
ADD mc_odo_change_t TIMESTAMP NULL DEFAULT NULL AFTER mc_odo;

This inserted a new field into the middle of a table with 227 columns.

This was resolved by killing and restarting the 'mysqld' node.

This table receives about 50 DML/sec from 12 mysqld nodes. Only one of them crashed.

Table schema and metrics below.

I hope this is useful to somebody.

Regards, Ben Clewett.

-----------------------------------------------------

Table Schema:

CREATE TABLE `fl_state` (
`terminal_id` varchar(32) NOT NULL,
`ip_addr` varchar(15) NOT NULL DEFAULT '',
`server_name` varchar(40) NOT NULL DEFAULT '',
`server_pid` int(11) NOT NULL DEFAULT '0',
`version_daemon` varchar(20) NOT NULL DEFAULT '',
`version_software` varchar(20) NOT NULL DEFAULT '',
`version_protocol` varchar(20) NOT NULL DEFAULT '',
`connected` timestamp NULL DEFAULT NULL,
`tally` int(11) NOT NULL DEFAULT '0',
`tacho_type` varchar(20) DEFAULT NULL,
`comp_code` varchar(8) DEFAULT NULL,
`veh_code` varchar(16) DEFAULT NULL,
`veh_rtid` int(11) DEFAULT NULL,
`mem_code` int(11) DEFAULT NULL,
`terminal_type` varchar(20) DEFAULT NULL,
`terminal_type_sub` varchar(20) DEFAULT NULL,
`terminal_type_sub_from_veh` varchar(20) DEFAULT NULL,
`service_code` varchar(16) DEFAULT NULL,
`service_sub_code` varchar(8) DEFAULT NULL,
`entity_type` enum('driven','trailer','member') NOT NULL DEFAULT 'driven',
`site_no` int(11) DEFAULT NULL,
`reset_required` tinyint(1) NOT NULL DEFAULT '0',
`tag_master_veh_rtid` int(11) DEFAULT NULL,
`tag_master_veh_rtid_prev` int(11) DEFAULT NULL,
`tag_master_terminal_id` varchar(32) DEFAULT NULL,
`tag_slave_recorded` tinyint(1) NOT NULL DEFAULT '0',
`tag_slave_count` int(11) NOT NULL DEFAULT '0',
`tag_state_pending` enum('trip','timeout','next_valid','not') NOT NULL DEFAULT 'not',
`tag_state_pending_t` timestamp NULL DEFAULT NULL,
`tag_state_pending_lat` decimal(9,6) DEFAULT NULL,
`tag_state_pending_lon` decimal(9,6) DEFAULT NULL,
`tag_state_pending_km_h` int(11) DEFAULT NULL,
`tag_state_pending_id` int(11) DEFAULT NULL,
`tag_state_timeout_start` timestamp NULL DEFAULT NULL,
`tag_state_timeout_t` int(11) DEFAULT NULL,
`tag_state_pending_trip` enum('wait_primed','wait_next','found_trip','not') NOT NULL DEFAULT 'not',
`tag_slave_last_drop` timestamp NULL DEFAULT NULL,
`tag_slave_last_hookup` timestamp NULL DEFAULT NULL,
`fl_story_rec` bigint(20) unsigned DEFAULT NULL,
`mo_t` timestamp NOT NULL DEFAULT '2000-01-01 00:00:00',
`mo_odo` decimal(11,3) DEFAULT NULL,
`mo_fuel` decimal(10,2) DEFAULT NULL,
`md_t` timestamp NOT NULL DEFAULT '2000-01-01 00:00:00',
`md_t_from` enum('gps','server') NOT NULL DEFAULT 'gps',
`md_valid_fix` tinyint(1) NOT NULL,
`md_ignition` tinyint(1) DEFAULT NULL,
`md_ignition_on_t` timestamp NULL DEFAULT NULL,
`md_ignition_off_t` timestamp NULL DEFAULT NULL,
`md_stationary` tinyint(1) DEFAULT NULL,
`md_lights` tinyint(1) DEFAULT NULL,
`md_battery` tinyint(1) DEFAULT NULL,
`md_panic_active` tinyint(1) DEFAULT NULL,
`md_panic_ack` tinyint(1) DEFAULT NULL,
`md_panic_active_t` timestamp NULL DEFAULT NULL,
`md_panic_cancel_t` timestamp NULL DEFAULT NULL,
`md_panic_ack_t` timestamp NULL DEFAULT NULL,
`md_stored_frame` tinyint(1) DEFAULT NULL,
`md_heading_change` tinyint(1) DEFAULT NULL,
`md_power_save` tinyint(1) DEFAULT NULL,
`md_gps_ariel_fault` enum('good','short','open') DEFAULT NULL,
`md_reset` tinyint(1) DEFAULT NULL,
`md_batt_volts` decimal(4,1) DEFAULT NULL,
`md_batt_percent` decimal(4,1) DEFAULT NULL,
`md_upgrade` tinyint(1) DEFAULT NULL,
`md_engine_by_volts` tinyint(1) DEFAULT NULL,
`md_flag1` tinyint(3) unsigned DEFAULT NULL,
`md_flag2` tinyint(3) unsigned DEFAULT NULL,
`md_tl_rec` bigint(20) unsigned DEFAULT NULL,
`md_server_name` varchar(40) DEFAULT NULL,
`mg_t` timestamp NULL DEFAULT NULL,
`mg_lat` decimal(9,6) DEFAULT NULL,
`mg_lon` decimal(9,6) DEFAULT NULL,
`mg_speed_n_h` decimal(5,1) DEFAULT NULL,
`mg_speed_km_h` decimal(5,1) DEFAULT NULL,
`mg_speed_m_h` decimal(5,1) DEFAULT NULL,
`mg_heading` int(11) DEFAULT NULL,
`mg_tl_rec` bigint(20) unsigned DEFAULT NULL,
`mg_server_name` varchar(40) DEFAULT NULL,
`mz_t` timestamp NULL DEFAULT NULL,
`mz_town` varchar(40) DEFAULT NULL,
`mz_county` varchar(40) DEFAULT NULL,
`mz_country` varchar(3) DEFAULT NULL,
`mz_pcode` varchar(16) DEFAULT NULL,
`mz_gaz_rec` int(11) DEFAULT NULL,
`mc_t` timestamp NULL DEFAULT NULL,
`mc_service_distance` int(11) DEFAULT NULL,
`mc_odo` decimal(11,3) DEFAULT NULL,
`mc_fuel` decimal(10,2) DEFAULT NULL,
`mc_fuel_level` decimal(4,1) DEFAULT NULL,
`mc_temperature` decimal(5,1) DEFAULT NULL,
`mc_cat_level` decimal(4,1) DEFAULT NULL,
`mc_axle_1` decimal(6,1) DEFAULT NULL,
`mc_axle_2` decimal(6,1) DEFAULT NULL,
`mc_axle_3` decimal(6,1) DEFAULT NULL,
`mc_axle_4` decimal(6,1) DEFAULT NULL,
`mc_flag1` tinyint(3) unsigned DEFAULT NULL,
`mc_flag2` tinyint(3) unsigned DEFAULT NULL,
`mc_engine_available` tinyint(1) DEFAULT NULL,
`mc_ending_on` tinyint(1) DEFAULT NULL,
`mc_engine_on` tinyint(1) DEFAULT NULL,
`mc_pto` tinyint(1) DEFAULT NULL,
`mc_from_trip` tinyint(1) DEFAULT NULL,
`mc_tl_rec` bigint(20) unsigned DEFAULT NULL,
`mc_server_name` varchar(40) DEFAULT NULL,
`ms_t` timestamp NULL DEFAULT NULL,
`ms_flag1` tinyint(3) unsigned DEFAULT NULL,
`ms_flag2` tinyint(3) unsigned DEFAULT NULL,
`ms_flag3` tinyint(3) unsigned DEFAULT NULL,
`ms_speed` decimal(5,1) DEFAULT NULL,
`ms_card_1` varchar(16) DEFAULT NULL,
`ms_card_2` varchar(16) DEFAULT NULL,
`ms_card_1_null` enum('not_null','unsupported_device','no_k_line','card_not_inserted','driver_unknown') DEFAULT NULL,
`ms_card_2_null` enum('not_null','unsupported_device','no_k_line','card_not_inserted','driver_unknown') DEFAULT NULL,
`ms_mem_code_1` int(11) DEFAULT NULL,
`ms_mem_code_2` int(11) DEFAULT NULL,
`ms_overspeed` tinyint(1) DEFAULT NULL,
`ms_time_state_1` enum('normal','15m_before_430h','430h','15m_before_900h','900h','15m_before_xxxh','xxxh') DEFAULT NULL,
`ms_time_state_2` enum('normal','15m_before_430h','430h','15m_before_900h','900h','15m_before_xxxh','xxxh') DEFAULT NULL,
`ms_card_1_mode` enum('w','r','d','a') DEFAULT NULL,
`ms_card_2_mode` enum('w','r','d','a') DEFAULT NULL,
`ms_tl_rec` bigint(20) unsigned DEFAULT NULL,
`ms_server_name` varchar(40) DEFAULT NULL,
`mt_t` timestamp NULL DEFAULT NULL,
`mt_trip_id` int(11) DEFAULT NULL,
`mt_source` enum('sdi1','sdi2','fc01') DEFAULT NULL,
`mt_trip_start` timestamp NULL DEFAULT NULL,
`mt_trip_end` timestamp NULL DEFAULT NULL,
`mt_trip_duration` int(11) DEFAULT NULL,
`mt_ignition_on_duration` int(11) DEFAULT NULL,
`mt_engine_on_duration` int(11) DEFAULT NULL,
`mt_motion_duration` int(11) DEFAULT NULL,
`mt_total_trip_duration` int(11) DEFAULT NULL,
`mt_pto_active_nomotion` int(11) DEFAULT NULL,
`mt_pto_active_motion` int(11) DEFAULT NULL,
`mt_trip_start_odometer` decimal(11,3) DEFAULT NULL,
`mt_trip_end_odometer` decimal(11,3) DEFAULT NULL,
`mt_trip_total_odometer` decimal(11,3) DEFAULT NULL,
`mt_trip_start_fuel` decimal(10,1) DEFAULT NULL,
`mt_trip_start_fuel_c` decimal(10,1) DEFAULT NULL,
`mt_trip_end_fuel` decimal(10,1) DEFAULT NULL,
`mt_trip_end_fuel_c` decimal(10,1) DEFAULT NULL,
`mt_trip_total_fuel` decimal(10,1) DEFAULT NULL,
`mt_trip_total_fuel_c` decimal(10,1) DEFAULT NULL,
`mt_trip_idle_fuel` decimal(10,1) DEFAULT NULL,
`mt_trip_idle_fuel_c` decimal(10,1) DEFAULT NULL,
`mt_trip_fuel_l_100km` decimal(6,1) DEFAULT NULL,
`mt_trip_fuel_l_100km_c` decimal(6,1) DEFAULT NULL,
`mt_trip_fuel_l_hr` decimal(6,1) DEFAULT NULL,
`mt_trip_fuel_l_hr_c` decimal(6,1) DEFAULT NULL,
`mt_trip_driving_duration` int(11) DEFAULT NULL,
`mt_trip_working_duration` int(11) DEFAULT NULL,
`mt_trip_rest_duration` int(11) DEFAULT NULL,
`mt_trip_available_duration` int(11) DEFAULT NULL,
`mt_trip_absolute_km_h` decimal(5,2) DEFAULT NULL,
`mt_trip_engine_on_km_h` decimal(5,2) DEFAULT NULL,
`mt_trip_motion_km_h` decimal(5,2) DEFAULT NULL,
`mt_trip_speed_zone1_cc_off_duration` int(11) DEFAULT NULL,
`mt_trip_speed_zone2_cc_off_duration` int(11) DEFAULT NULL,
`mt_trip_speed_zone3_cc_off_duration` int(11) DEFAULT NULL,
`mt_trip_speed_zone4_cc_off_duration` int(11) DEFAULT NULL,
`mt_trip_speed_total_cc_off_duration` int(11) DEFAULT NULL,
`mt_trip_speed_zone1_cc_on_duration` int(11) DEFAULT NULL,
`mt_trip_speed_zone2_cc_on_duration` int(11) DEFAULT NULL,
`mt_trip_speed_zone3_cc_on_duration` int(11) DEFAULT NULL,
`mt_trip_speed_zone4_cc_on_duration` int(11) DEFAULT NULL,
`mt_trip_speed_total_cc_on_duration` int(11) DEFAULT NULL,
`mt_trip_brake_pedal_count` int(11) DEFAULT NULL,
`mt_trip_brake_pedal_duration` int(11) DEFAULT NULL,
`mt_trip_accel_duration` int(11) DEFAULT NULL,
`mt_trip_decel_duration` int(11) DEFAULT NULL,
`mt_trip_accel_max_ms2` decimal(5,2) DEFAULT NULL,
`mt_trip_decel_max_ms2` decimal(5,2) DEFAULT NULL,
`mt_trip_acc_zone1_duration` int(11) DEFAULT NULL,
`mt_trip_acc_zone2_duration` int(11) DEFAULT NULL,
`mt_trip_acc_zone3_duration` int(11) DEFAULT NULL,
`mt_trip_acc_zone4_duration` int(11) DEFAULT NULL,
`mt_trip_acc_zone5_duration` int(11) DEFAULT NULL,
`mt_trip_acc_zone6_duration` int(11) DEFAULT NULL,
`mt_trip_acc_zone7_duration` int(11) DEFAULT NULL,
`mt_trip_throttle_zone1_duration` int(11) DEFAULT NULL,
`mt_trip_throttle_zone2_duration` int(11) DEFAULT NULL,
`mt_trip_throttle_zone3_duration` int(11) DEFAULT NULL,
`mt_trip_throttle_zone4_duration` int(11) DEFAULT NULL,
`mt_trip_engine_zone1_duration` int(11) DEFAULT NULL,
`mt_trip_engine_zone2_duration` int(11) DEFAULT NULL,
`mt_trip_engine_zone3_duration` int(11) DEFAULT NULL,
`mt_message_type` enum('T','U') DEFAULT NULL,
`mt_time_engine_torque_high` int(11) DEFAULT NULL,
`mt_time_in_high_gear` int(11) DEFAULT NULL,
`mt_time_high_engine_load` int(11) DEFAULT NULL,
`mt_time_low_load_high_throttle` int(11) DEFAULT NULL,
`mt_time_kick_down` int(11) DEFAULT NULL,
`mt_mean_engine_load` decimal(4,1) DEFAULT NULL,
`mt_server_name` varchar(40) DEFAULT NULL,
`created` timestamp NOT NULL DEFAULT '2000-01-01 00:00:00',
`updated` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
`ignore_pto` tinyint(1) NOT NULL DEFAULT '0',
`cfg_t` timestamp NULL DEFAULT NULL,
`cfg_apn` varchar(20) DEFAULT NULL,
`cfg_user` varchar(20) DEFAULT NULL,
`cfg_password` varchar(20) DEFAULT NULL,
`cfg_ignition_off_every` varchar(20) DEFAULT NULL,
`cfg_hd_change_rpt` varchar(20) DEFAULT NULL,
`cfg_can_fitted` varchar(20) DEFAULT NULL,
`cfg_extended_can` varchar(20) DEFAULT NULL,
`cfg_trip_every` varchar(20) DEFAULT NULL,
`cfg_tacho_download_enabled` varchar(20) DEFAULT NULL,
`cfg_can_comms_protocol` varchar(20) DEFAULT NULL,
`cfg_network_sel` varchar(20) DEFAULT NULL,
`cfg_ign_rep` varchar(20) DEFAULT NULL,
`cfg_debug` varchar(20) DEFAULT NULL,
`cfg_every` varchar(20) DEFAULT NULL,
`cfg_on_battery_backup_every` varchar(20) DEFAULT NULL,
`cfg_ign_sen` varchar(20) DEFAULT NULL,
`cfg_auto_reset` varchar(20) DEFAULT NULL,
`cfg_red_pwr_mod` varchar(20) DEFAULT NULL,
`cfg_red_pwr_min` varchar(20) DEFAULT NULL,
`cfg_eng_sense_enabled` varchar(20) DEFAULT NULL,
`cfg_eng_on_offset` varchar(20) DEFAULT NULL,
`cfg_panic_enabled` varchar(20) DEFAULT NULL,
`cfg_logging` varchar(20) DEFAULT NULL,
`cfg_rmt_logging` varchar(20) DEFAULT NULL,
`cfg_rmt_log_period` varchar(20) DEFAULT NULL,
`cfg_reset_fuel` varchar(20) DEFAULT NULL,
`cfg_alarm_time` varchar(20) DEFAULT NULL,
`cnf_auto_request` enum('no','yes_new','yes_done') NOT NULL DEFAULT 'no',
`cnf_auto_request_t` timestamp NULL DEFAULT NULL,
PRIMARY KEY (`terminal_id`),
KEY `comp_code` (`comp_code`,`veh_code`),
KEY `veh_rtid` (`veh_rtid`),
KEY `tag_master_terminal_id` (`tag_master_terminal_id`),
KEY `tag_master_veh_rtid` (`tag_master_veh_rtid`),
KEY `cnf_auto_request` (`cnf_auto_request`,`terminal_type_sub`)
) ENGINE=ndbcluster DEFAULT CHARSET=latin1



-----------------------------------------------------

Information Schema:

TABLE_CATALOG: def
TABLE_SCHEMA: ndb
TABLE_NAME: fl_state
TABLE_TYPE: BASE TABLE
ENGINE: ndbcluster
VERSION: 10
ROW_FORMAT: Dynamic
TABLE_ROWS: 6567
AVG_ROW_LENGTH: 844
DATA_LENGTH: 7405568
MAX_DATA_LENGTH: 0
INDEX_LENGTH: 0
DATA_FREE: 0
AUTO_INCREMENT: NULL
CREATE_TIME: NULL
UPDATE_TIME: NULL
CHECK_TIME: NULL
TABLE_COLLATION: latin1_swedish_ci
CHECKSUM: NULL
CREATE_OPTIONS:
TABLE_COMMENT:

Has anyone created a MySQL Cluster with InnoDB and MyISAM tables.

We have a MySQL database having 30% tables of MyISAM and 70% tables of InnoDB. Now we want to migrate this on MySQL cluster.

If anyone has been already done something like this Please tell me the steps.