HammerDB 5.0 TPROC-H with MariaDB/ColumnStore - Errors with multiple tpch_threads

[AspFTM]

Hello Steve,

I am exploring HammerDB Analytic testing capabilities with MariaDB ColumnStore, and running into some problems.

My Setup:
2 Socket Intel Xeon server with 96Cores/192Threads and 768GB Memory
OS - Ubuntu 24.04 LTS with Linux 6.8
MariaDB comunity server v11.8.5 along with ColumnStore, installed directly from Mariadb (not compiled locally)
HammerDB 5.0

MariaDB server and HammerDB on the same system. My problem is as follows:

1. Using the included buildschema script for TPROC-H for MariaDB with maria_tpch_storage_engine set to Columnstore and maria_scale_fact 1, I can get the schema to build correctly - takes less than an hour. I even got the schema to build with scale factor 10, although it took many hours. Afterwards I can use the run script with Columstore, to get query results.

2. However, if I increase maria_num_tpch_threads to more than 1, even 2 or 4, then the schema build has errors. The errors show up immediately after the start of the build. Here is an example of the errors:

Vuser 5:Loading SUPPLIER...
Error in Virtual User 1: maria::exec/db server: Internal error: CAL0001: Insert Failed: Lost connection to Write Engine Server while updating SYSTABLES
Vuser 1:FINISHED FAILED
Error in Virtual User 3: maria::exec/db server: Internal error: Lost connection to DMLProc [4]
Vuser 3:FINISHED FAILED
Error in Virtual User 4: maria::exec/db server: Internal error: Lost connection to DMLProc [4]
Vuser 4:FINISHED FAILED
Error in Virtual User 2: maria::exec/db server: Internal error: Lost connection to DMLProc [4]
Vuser 2:FINISHED FAILED
Error in Virtual User 5: maria::exec/db server: Internal error: Lost connection to DMLProc after getting a new client [2:InetStreamSocket::connect: connect() error: Connection refused to: InetStreamSocket: sd: 64 inet: 127.0.0.1 port: 8614 then InetStreamS
Vuser 5:FINISHED FAILED
ALL VIRTUAL USERS COMPLETE
TPROC-H Driver Script
SCHEMA BUILD COMPLETED

In this specific run, all 4 threads have errors and the build immediately ended. Sometimes some threads do not have the 'lost connection errors' and build completes, but I suspect the schema is not fully built in those runs. So the result is not useful.

In /var/log/mariadb/columnstore/debug.log I see errors when the failing schema build is run:

threadpool[1525063]: 26.121008 |0|0|0| E 22 CAL0005: ThreadPool: Caught exception during execution: MCS-2064: Out of memory.
threadpool[1525063]: 26.121047 |0|0|0| E 22 CAL0005: ThreadPool: Caught exception during execution: MCS-2064: Out of memory.
..
dmlpackageproc[1525069]: 26.352831 |0|0|0| E 21 CAL0001: Insert Failed: Lost connection to Write Engine Server while updating SYSTABLES
messagequeue[1524456]: 26.510277 |0|0|0| W 31 CAL0000: Client read close socket for InetStreamSocket::readToMagic(): I/O error2.1: err = -1 e = 104: Connection reset by peer %%10%%

I have used "optimized linux kernel parameters" as suggested by MariaDB/Columnstore installation page here -
https://mariadb.com/docs/server/architecture/topologies/single-node-topologies/community-server-with-columnstore

I also tried to use Huge pages, it did not make any difference. I get the same errors.

In MariadB Server configuration file, I have set a few timeouts to really high values - 1000, 2000 etc. and max connections to few hundred thousand. Other than that I have not done any MariaDB configuration.
From Columnstore McsGetConfig, HashJoin.TotalUmMemory is set to 65%. I reduced this to 10%, to no effect.

Now Innodb engine when used for TPROC-H has no problem with schema build using large number of threads. But I would like to get Columnstore schema built for much larger scale factor (100 or up). That doesn't seem feasible if I can't use multiple threads to build schema and speedup the buid.

My question - Is there anything I am missing, some MariaDB/ColumnStore option or HammerDB option? Or is this a problem with Columnstore and needs to be reported there?

Thanks.

[sm-shaw]

OK, there are a couple of things here:
Firstly, inserting directly into a columnstore is not generally advisable, the reason is we are inserting in row format and this is being translated into column format, you see an error mentioning the Write Engine Server which is what is doing this, it is not entirely clear how scalable this is?
So if you want to test a columnstore then the recommendation is to use the datagen feature to do a bulk load https://www.hammerdb.com/docs/ch13.html, you will save a lot of time in the long run doing this. There is a guide here on how to do this after HammerDB has created the files: https://mariadb.org/wp-content/uploads/2025/02/Discover-what-columnstore-can-really-do-for-you.pdf
Secondly, having said this you wouldn't expect things to fall over with 2 threads doing inserts, this does not look from the evidence as a HammerDB issue and is instead a MariaDB columnstore server issue, potentially with configuration because as you say and as I can confirm with InnoDB there is no insert issue with multiple threads as you would expect, but using the guide above is the best way to create the schema.

[AspFTM]

Thanks, creating data files with HammerDB datagen and then importing those files with MariaDB cpimport as described in the MariaDB document worked. I did change the Date formats in lineitem and orders files before cpimport to make it compatible with MariaDB date format.