I am running 4 nodes. All separate servers.
2 mgmt nodes.
2 data/sql nodes.
The problem I have is this.
When all the nodes are running. I telnet into memcached and do the following:
[root@db ~]# telnet localhost 11211
Trying ::1...
Connected to localhost.
Escape character is '^]'.
set key 0 900 4
data
STORED
quit
Connection closed by foreign host.
I then check memcached on the node and I see the process running at 100%, pretty much immediately I do the set command. The box is pretty much toast at this point because doing queries is like running through sludge. I have to kill memcached to get the box back. :(
The top result:
2345 root 20 0 1126412 16428 3124 S **103.1** 0.0 0:46.51 memcached
This even happens when I do a simple get key which returns the value. Again, the memcached process shoots up from %5 to 100%.
Can someone please tell me why this is?
I'm running CentOS 7 on all the servers, with a minimal installation.
I'm running Cluster 7.3.7 and I have even tested with 7.4.2, same issue.
This was a clean install, mysql-cluster is very fresh, no DB installed!
Here is my config.ini for the mgmt nodes:
[ndb_mgmd default]
datadir=/var/lib/mysql-cluster # Directory for MGM node log files
[ndb_mgmd]
# Management process options:
hostname=127.0.0.232 # Hostname or IP address of MGM node
NodeId=101
#[ndb_mgmd]
# Management process options:
hostname=127.0.0.235 # Hostname or IP address of MGM node
NodeId=102
[ndbd default]
# Options affecting ndbd processes on all data nodes:
NoOfReplicas=1 # Number of replicas
datadir=/var/lib/mysql-cluster/data # Directory for this data node's data files
DataMemory=10G # How much memory to allocate for data storage
IndexMemory=2G # How much memory to allocate for index storage
LockPagesInMainMemory=1 # Locks data node processes into memory. Doing so prevents them from swapping to disk
MaxNoOfConcurrentOperations=1000000
MaxNoOfTables=200
MaxNoOfAttributes=4000
MaxNoOfOrderedIndexes=10000
[ndbd]
# Options for data node "A":
# (one [ndbd] section per data node)
hostname=127.0.0.233 # Hostname or IP address
NodeId=1
#[ndbd]
# Options for data node "B":
hostname=127.0.0.236 # Hostname or IP address
NodeId=2
[mysqld]
# SQL node options:
hostname=127.0.0.233 # Hostname or IP address
NodeId=51
[mysqld]
# SQL node options (1st for memcached):
hostname=127.0.0.233 # Hostname or IP address
[mysqld]
# SQL node options (2nd for memcached):
hostname=127.0.0.233 # Hostname or IP address
[mysqld]
# SQL node options (3rd for memcached):
hostname=127.0.0.233 # Hostname or IP address
[mysqld]
# SQL node options (4th for memcached):
hostname=127.0.0.233 # Hostname or IP address
#[mysqld]
# SQL node options:
hostname=127.0.0.236 # Hostname or IP address
NodeId=52
#[mysqld]
# SQL node options (1st for memcached):
hostname=127.0.0.236 # Hostname or IP address
#[mysqld]
# SQL node options (2nd for memcached):
hostname=127.0.0.236 # Hostname or IP address
#[mysqld]
# SQL node options (3rd for memcached):
hostname=127.0.0.236 # Hostname or IP address
#[mysqld]
# SQL node options (4th for memcached):
hostname=127.0.0.236 # Hostname or IP address
Here is my mysql.cnf for the sql node:
[mysqld]
# Options for mysqld process:
ndbcluster # run NDB storage engine
ndb-nodeid=51
server-id=51
[mysql_cluster]
# Options for MySQL Cluster processes:
ndb-connectstring=127.0.0.232 # location of management server
[mysql]
prompt="(\u@\h) [\d]>\_"
Here is my show output: (Don't worry that I am only showing you one half). I've even tried just 1 half of the setup to see if I can still replicate it and I can. I've tried even with 3 mgmt nodes, 3 sql/data nodes and still the same result. /sigh
It's midnight here and I'm too tired to get the other half up just to show you the correct show result. Trust me, it shows correctly all the nodes.
ndb_mgm> show
Cluster Configuration
---------------------
[ndbd(NDB)] 1 node(s)
id=1 @127.0.0.233 (mysql-5.6.21 ndb-7.3.7, Nodegroup: 0, *)
[ndb_mgmd(MGM)] 1 node(s)
id=101 @127.0.0.232 (mysql-5.6.21 ndb-7.3.7)
[mysqld(API)] 5 node(s)
id=51 @127.0.0.233 (mysql-5.6.21 ndb-7.3.7)
id=52 @127.0.0.233 (mysql-5.6.21 ndb-7.3.7)
id=53 @127.0.0.233 (mysql-5.6.21 ndb-7.3.7)
id=54 @127.0.0.233 (mysql-5.6.21 ndb-7.3.7)
id=55 (not connected, accepting connect from 127.0.0.233)
Here is how I'm installing and running memcached:
mysql -u root -pPASSWORD < /usr/share/mysql/memcache-api/ndb_memcache_metadata.sql
memcached -u root -E /usr/lib64/ndb_engine.so -e "connectstring=127.0.0.232:1186;role=db-only" -vv -c 20
Any in-sights to this really appreciated! I will attempt in trying lower versions of mysql cluster. I may even try 7.4 which is in beta... I really need this to work.
I don't have this problem at all when I run with innodb-memcached nor memcached server standalone. Just letting you know, that I have experience with other memcached installations.
Thanks
Paul
2 mgmt nodes.
2 data/sql nodes.
The problem I have is this.
When all the nodes are running. I telnet into memcached and do the following:
[root@db ~]# telnet localhost 11211
Trying ::1...
Connected to localhost.
Escape character is '^]'.
set key 0 900 4
data
STORED
quit
Connection closed by foreign host.
I then check memcached on the node and I see the process running at 100%, pretty much immediately I do the set command. The box is pretty much toast at this point because doing queries is like running through sludge. I have to kill memcached to get the box back. :(
The top result:
2345 root 20 0 1126412 16428 3124 S **103.1** 0.0 0:46.51 memcached
This even happens when I do a simple get key which returns the value. Again, the memcached process shoots up from %5 to 100%.
Can someone please tell me why this is?
I'm running CentOS 7 on all the servers, with a minimal installation.
I'm running Cluster 7.3.7 and I have even tested with 7.4.2, same issue.
This was a clean install, mysql-cluster is very fresh, no DB installed!
Here is my config.ini for the mgmt nodes:
[ndb_mgmd default]
datadir=/var/lib/mysql-cluster # Directory for MGM node log files
[ndb_mgmd]
# Management process options:
hostname=127.0.0.232 # Hostname or IP address of MGM node
NodeId=101
#[ndb_mgmd]
# Management process options:
hostname=127.0.0.235 # Hostname or IP address of MGM node
NodeId=102
[ndbd default]
# Options affecting ndbd processes on all data nodes:
NoOfReplicas=1 # Number of replicas
datadir=/var/lib/mysql-cluster/data # Directory for this data node's data files
DataMemory=10G # How much memory to allocate for data storage
IndexMemory=2G # How much memory to allocate for index storage
LockPagesInMainMemory=1 # Locks data node processes into memory. Doing so prevents them from swapping to disk
MaxNoOfConcurrentOperations=1000000
MaxNoOfTables=200
MaxNoOfAttributes=4000
MaxNoOfOrderedIndexes=10000
[ndbd]
# Options for data node "A":
# (one [ndbd] section per data node)
hostname=127.0.0.233 # Hostname or IP address
NodeId=1
#[ndbd]
# Options for data node "B":
hostname=127.0.0.236 # Hostname or IP address
NodeId=2
[mysqld]
# SQL node options:
hostname=127.0.0.233 # Hostname or IP address
NodeId=51
[mysqld]
# SQL node options (1st for memcached):
hostname=127.0.0.233 # Hostname or IP address
[mysqld]
# SQL node options (2nd for memcached):
hostname=127.0.0.233 # Hostname or IP address
[mysqld]
# SQL node options (3rd for memcached):
hostname=127.0.0.233 # Hostname or IP address
[mysqld]
# SQL node options (4th for memcached):
hostname=127.0.0.233 # Hostname or IP address
#[mysqld]
# SQL node options:
hostname=127.0.0.236 # Hostname or IP address
NodeId=52
#[mysqld]
# SQL node options (1st for memcached):
hostname=127.0.0.236 # Hostname or IP address
#[mysqld]
# SQL node options (2nd for memcached):
hostname=127.0.0.236 # Hostname or IP address
#[mysqld]
# SQL node options (3rd for memcached):
hostname=127.0.0.236 # Hostname or IP address
#[mysqld]
# SQL node options (4th for memcached):
hostname=127.0.0.236 # Hostname or IP address
Here is my mysql.cnf for the sql node:
[mysqld]
# Options for mysqld process:
ndbcluster # run NDB storage engine
ndb-nodeid=51
server-id=51
[mysql_cluster]
# Options for MySQL Cluster processes:
ndb-connectstring=127.0.0.232 # location of management server
[mysql]
prompt="(\u@\h) [\d]>\_"
Here is my show output: (Don't worry that I am only showing you one half). I've even tried just 1 half of the setup to see if I can still replicate it and I can. I've tried even with 3 mgmt nodes, 3 sql/data nodes and still the same result. /sigh
It's midnight here and I'm too tired to get the other half up just to show you the correct show result. Trust me, it shows correctly all the nodes.
ndb_mgm> show
Cluster Configuration
---------------------
[ndbd(NDB)] 1 node(s)
id=1 @127.0.0.233 (mysql-5.6.21 ndb-7.3.7, Nodegroup: 0, *)
[ndb_mgmd(MGM)] 1 node(s)
id=101 @127.0.0.232 (mysql-5.6.21 ndb-7.3.7)
[mysqld(API)] 5 node(s)
id=51 @127.0.0.233 (mysql-5.6.21 ndb-7.3.7)
id=52 @127.0.0.233 (mysql-5.6.21 ndb-7.3.7)
id=53 @127.0.0.233 (mysql-5.6.21 ndb-7.3.7)
id=54 @127.0.0.233 (mysql-5.6.21 ndb-7.3.7)
id=55 (not connected, accepting connect from 127.0.0.233)
Here is how I'm installing and running memcached:
mysql -u root -pPASSWORD < /usr/share/mysql/memcache-api/ndb_memcache_metadata.sql
memcached -u root -E /usr/lib64/ndb_engine.so -e "connectstring=127.0.0.232:1186;role=db-only" -vv -c 20
Any in-sights to this really appreciated! I will attempt in trying lower versions of mysql cluster. I may even try 7.4 which is in beta... I really need this to work.
I don't have this problem at all when I run with innodb-memcached nor memcached server standalone. Just letting you know, that I have experience with other memcached installations.
Thanks
Paul