Nagios reporting /dev/shm full on data nodes (2 replies)

Hi,

I've got a cluster setup with a management node and two data nodes. Recently the nagios server which is set up to monitor disk space (amongst other things) started warning about the /dev/shm filesystem being 100% full. This warning would "flap" i.e. go from a warning state to a non-warning state.

After logging into this machine and running df, it showed that this filesystem is completely empty:

# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                       26G   20G  4.7G  81% /
/dev/xvda1             99M   43M   52M  46% /boot
tmpfs                  10G     0   10G   0% /dev/shm

The only software running on this machine is ndbd. Also, I should mention that nagios has never reported this error before, although recently I have been migrating a lot of data into to the cluster tables. I have stopped inserting any more data into the cluster tables, but the warning remains.

My thought is that the data nodes may be running out of memory, but after checking the memory usage, it doesn't seem like it:

ndb_mgm> 3 REPORT MemoryUsage
Node 3: Index usage is 13%(24473 8K pages of total 180384)
Node 3: Data usage is 43%(155534 32K pages of total 360512)

ndb_mgm> 2 REPORT MemoryUsage
Node 2: Index usage is 13%(24473 8K pages of total 180384)
Node 2: Data usage is 43%(155536 32K pages of total 360512)

BTW, how do I read the above numbers? i.e. are they both using 1.8G for indexes and 3.6G of memory for data? The data nodes each have 20GB of memory in total and DataMemory = 11266M, IndexMemory = 1409M in config.ini.

The version of Cluster that I'm using is:

MySQL distrib mysql-5.1.39 ndb-7.0.9b, for unknown-linux-gnu (x86_64)

and the OS it's running on is CentOS release 5.4 (Final) 64-bit version