Disk storage blob using more memory than expected. (1 reply)

Hi All,

I'm evaluating Mysql Cluster, but having a problem using disk storage and blobs.

I have read the documentation that says the first 256 bytes of a blob will be held in DataMemory for disk based storage, but when I tried adding various sizes of blobs they started using different amounts of DataMemory.

To test I configured a very basic cluster, and used the following setup:

CREATE DATABASE test;
USE TEST;

DELIMITER //
CREATE PROCEDURE GenerateFakeData(in_records int, in_strlen int)
BEGIN
DECLARE i INT DEFAULT 1;
SET @var = lpad('', in_strlen, sha1(rand()));
WHILE i <= in_records DO
INSERT INTO testblob (id, data) VALUES (i, @var);
SET i = i + 1;
END WHILE;
END //
DELIMITER ;

CREATE LOGFILE GROUP lg_01
ADD UNDOFILE 'undofile.dat' ENGINE = NDB;

CREATE TABLESPACE ts_01
ADD DATAFILE 'datafile.dat'
USE LOGFILE GROUP lg_01
INITIAL_SIZE = 4096M ENGINE = NDB;

CREATE TABLE `testblob` (
`id` int(11) NOT NULL,
`data` longblob NOT NULL STORAGE DISK,
PRIMARY KEY (`id`)
)
TABLESPACE ts_01 STORAGE DISK
ENGINE = NDB;


Now for the tests:

mysql> truncate table testblob;
Query OK, 0 rows affected (0.21 sec)

mysql> select * from ndbinfo.memoryusage;
+---------+---------------------+----------+------------+----------+-------------+
| node_id | memory_type | used | used_pages | total | total_pages |
+---------+---------------------+----------+------------+----------+-------------+
| 1 | Data memory | 12484608 | 381 | 83886080 | 2560 |
| 1 | Index memory | 688128 | 84 | 19136512 | 2336 |
| 1 | Long message buffer | 393216 | 1536 | 67108864 | 262144 |
| 2 | Data memory | 12451840 | 380 | 83886080 | 2560 |
| 2 | Index memory | 688128 | 84 | 19136512 | 2336 |
| 2 | Long message buffer | 393216 | 1536 | 67108864 | 262144 |
+---------+---------------------+----------+------------+----------+-------------+
6 rows in set (0.01 sec)

mysql> CALL GenerateFakeData(15000, 10240);
Query OK, 1 row affected (32.36 sec)

mysql> select count(id) from testblob;
+-----------+
| count(id) |
+-----------+
| 15000 |
+-----------+
1 row in set (0.01 sec)

mysql> select * from ndbinfo.memoryusage;
+---------+---------------------+----------+------------+----------+-------------+
| node_id | memory_type | used | used_pages | total | total_pages |
+---------+---------------------+----------+------------+----------+-------------+
| 1 | Data memory | 18186240 | 555 | 83886080 | 2560 |
| 1 | Index memory | 1130496 | 138 | 19136512 | 2336 |
| 1 | Long message buffer | 262144 | 1024 | 67108864 | 262144 |
| 2 | Data memory | 18186240 | 555 | 83886080 | 2560 |
| 2 | Index memory | 1130496 | 138 | 19136512 | 2336 |
| 2 | Long message buffer | 393216 | 1536 | 67108864 | 262144 |
+---------+---------------------+----------+------------+----------+-------------+
6 rows in set (0.01 sec)

mysql> truncate table testblob;
Query OK, 0 rows affected (0.18 sec)

mysql> select * from ndbinfo.memoryusage;
+---------+---------------------+----------+------------+----------+-------------+
| node_id | memory_type | used | used_pages | total | total_pages |
+---------+---------------------+----------+------------+----------+-------------+
| 1 | Data memory | 12484608 | 381 | 83886080 | 2560 |
| 1 | Index memory | 688128 | 84 | 19136512 | 2336 |
| 1 | Long message buffer | 262144 | 1024 | 67108864 | 262144 |
| 2 | Data memory | 12451840 | 380 | 83886080 | 2560 |
| 2 | Index memory | 688128 | 84 | 19136512 | 2336 |
| 2 | Long message buffer | 262144 | 1024 | 67108864 | 262144 |
+---------+---------------------+----------+------------+----------+-------------+
6 rows in set (0.01 sec)

mysql> CALL GenerateFakeData(15000, 102400);
Query OK, 1 row affected (2 min 24.93 sec)

mysql> select count(id) from testblob;
+-----------+
| count(id) |
+-----------+
| 15000 |
+-----------+
1 row in set (0.00 sec)

mysql> select * from ndbinfo.memoryusage;
+---------+---------------------+----------+------------+----------+-------------+
| node_id | memory_type | used | used_pages | total | total_pages |
+---------+---------------------+----------+------------+----------+-------------+
| 1 | Data memory | 21954560 | 670 | 83886080 | 2560 |
| 1 | Index memory | 2711552 | 331 | 19136512 | 2336 |
| 1 | Long message buffer | 393216 | 1536 | 67108864 | 262144 |
| 2 | Data memory | 21954560 | 670 | 83886080 | 2560 |
| 2 | Index memory | 2711552 | 331 | 19136512 | 2336 |
| 2 | Long message buffer | 131072 | 512 | 67108864 | 262144 |
+---------+---------------------+----------+------------+----------+-------------+
6 rows in set (0.01 sec)


So adding 15,000 records with a blob of 10k the DataMemory used was 18,186,240, but when I inserted 15,000 100k blobs the DataMemory used was 21,954,560.

The table is configured as disk storage, so the amount of DataMemory used should not be dependent on the size of the blob.

Where am I going wrong, and is there a way to query what is inside DataMemory ?

Any help will be apreciated.