So my first experience with MySQL cluster/NDB has been far less than stellar.
I have a corrupt database, which has never happened with InnoDB.
Set up and ran a cluster, low volume, NoCopies=2
1 x NDB + mysql on metal where all clients connected
1 x NDB + mysql on Oracle VirtualBox
1 x mnmt on secondary Oracle VirtualBox
During the course of operation, I got some table read errors, Err 233. (No text error message associated.) Looked it up, very suspicious that I was hitting the 32k MaxNoOfConcurrentOperations limit, but moved it up anyways to 65535.
Eventually ran out of disk on the NDB+mysql VM. Shut everything down, restarted with another VM disk attached for data and bumped memory alloc while it was down.
Still getting MaxNoOfConcurrentOperations-type errors, but fewer, even at 65535. This is dumb because we're sub 20 (TWENTY!) TPS with almost all of them being reads. No way it should be hitting 32k concurrent, much less 64k. These showed up on the bare-metal box, not the VM box.
Looking at the database, it's showing corruption ... it looks like some of transactions were completely replayed... the UPDATE calls which used "col = col + x" are all skewed. (Note: I did not tell it to use binlog or =row.)
1) Why would I possibly be getting Err: 233 off such a stupid low number of twenty TPS on the bare-metal box? I could see the VM potentially having issues, but it was literally just a replication target in the cluster.
2) Why would I end up with getting replayed statement transactions? How can I even start to trace this? How does this even happen? And importantly, how can I ensure that this never (ever) happens again?
Thanks ... hopefully someone can shed some light on this!
I have a corrupt database, which has never happened with InnoDB.
Set up and ran a cluster, low volume, NoCopies=2
1 x NDB + mysql on metal where all clients connected
1 x NDB + mysql on Oracle VirtualBox
1 x mnmt on secondary Oracle VirtualBox
During the course of operation, I got some table read errors, Err 233. (No text error message associated.) Looked it up, very suspicious that I was hitting the 32k MaxNoOfConcurrentOperations limit, but moved it up anyways to 65535.
Eventually ran out of disk on the NDB+mysql VM. Shut everything down, restarted with another VM disk attached for data and bumped memory alloc while it was down.
Still getting MaxNoOfConcurrentOperations-type errors, but fewer, even at 65535. This is dumb because we're sub 20 (TWENTY!) TPS with almost all of them being reads. No way it should be hitting 32k concurrent, much less 64k. These showed up on the bare-metal box, not the VM box.
Looking at the database, it's showing corruption ... it looks like some of transactions were completely replayed... the UPDATE calls which used "col = col + x" are all skewed. (Note: I did not tell it to use binlog or =row.)
1) Why would I possibly be getting Err: 233 off such a stupid low number of twenty TPS on the bare-metal box? I could see the VM potentially having issues, but it was literally just a replication target in the cluster.
2) Why would I end up with getting replayed statement transactions? How can I even start to trace this? How does this even happen? And importantly, how can I ensure that this never (ever) happens again?
Thanks ... hopefully someone can shed some light on this!