Slow building of GTDB database

[larssnip]

Hi,

Thanks for this tool, this is something I have looked forward to.

But, I have some questions. I have downloaded the pre-built GTDB database, and it works fine. However, since it is release 2.14 I want the newest (r2.26). I started building it, but it takes a loooong time!

After the initial checking, metabuli starts to fill a temporary folder with unzipped fasta files, the genomes, where the file names are the accession number. This processes around 6-7 genomes per minute. With 143 000 genomes this will take weeks! I have used 10 threads.

What exactly is done during this processing? It looks to me like it only copies the genome files and decompress them? The log-file does not say anything. Is there any way I can speed this up? There is a mentioning of CDS-files. Does this have to be files downloaded from NCBI (refseg/genbank)? Or could I just run prodigal myself (with many array jobs) to do this job in no time?

LS

[jaebeom-kim]

Hi! Thank you for reaching out.
You can skip the process using --make-library 0.
It is set as 1 by default, but it is okay to skip for your case.
It is needed when using a single large FASTA file including multiple species.
For CDS, you can download them from refseq/genbank.
Please download files suffixed with cds_from_genomic.fna.gz

I'm working on a new indexing method to make a 2X smaller DB.
Since a DB of 143,000 genomes is expected to be large as 1.2 TB, making a 2X smaller DB would benefit a lot.
Could you please test it when ready?

Best regards,
Jaebeom

[larssnip]

Hi,

Thanks for the hint, I have now started a build with --make-library 0.

But, I suggest you update the text on your GitHub site accordingly. First, under GTDB-based custom database this option is not listed. Also, since it assumes the user has a FASTA_LIST i suppose the situation will always look like mine here? It would make sense to me to list this option, and also use it in the example command line. Then, further down, under NCBI or custom taxonomy database the option --make-library is listed, but it says the default is 0 (not 1 as you write above).

I have downloaded the CDS files from NCBI, but found only the RefSeq part of them really have such files. The Genbank genomes rarely have them. Also, a problem with GTDB is that the genomes they use when a new version is created is a snapshot of NCBI at one certain day (?). The GenBank and RefSeq changes every day, and of the 143 000 genomes in GTDB r226 only 101 000 are now listed at NCBI according to the accession numbers. Thus, I found this downloading of CDS files of limited use since I could download files for only around 25 000 out of 143 000 genomes.

Do you have any comments on this?

[larssnip]

Hi again,

Sounds good with your attempt to make a reduced database! Are you talking about the disk space or the memory (RAM) used when using it? I can see the default memory usage is 128G in metabuli. I have built a kraken database of the GTDB r226 genomes, and it needs 1T of memory (RAM) when used. Will this be the same for metabuli?

[jaebeom-kim]

But, I suggest you update the text on your GitHub site accordingly. First, under GTDB-based custom database this option is not listed. Also, since it assumes the user has a FASTA_LIST i suppose the situation will always look like mine here? It would make sense to me to list this option, and also use it in the example command line. Then, further down, under NCBI or custom taxonomy database the option --make-library is listed, but it says the default is 0 (not 1 as you write above).
I meant the disk space.

Sorry for this confusion, I will improve the manual soon.

I have downloaded the CDS files from NCBI, but found only the RefSeq part of them really have such files. The Genbank genomes rarely have them.

You don't have to provide CDS. It is actually mainly for eukaryote, for which Prodigal cannot predict genes correctly.
As far as I know, CDS files from NCBI are generated using Prodigal in many cases.

Also, a problem with GTDB is that the genomes they use when a new version is created is a snapshot of NCBI at one certain day (?). The GenBank and RefSeq changes every day, and of the 143 000 genomes in GTDB r226 only 101 000 are now listed at NCBI according to the accession numbers.

I recommend using assembly accessions without version number.
For GCA_1234578.4, just use GCA_12345678.
Maybe this script helps you download_gtdb_hq_genomes.sh. It includes a quality-filtering step, so you may want to skip the filtering codes.

Sounds good with your attempt to make a reduced database! Are you talking about the disk space or the memory (RAM) used when using it? I can see the default memory usage is 128G in metabuli. I have built a kraken database of the GTDB r226 genomes, and it needs 1T of memory (RAM) when used. Will this be the same for metabuli?

Metabuli doesn't load the database on RAM. It is designed to utilize user-specified RAM size.
This feature with execution times on Linux server, Windows desktop, Macbook are provided in this preprint.
https://www.biorxiv.org/content/10.1101/2025.03.10.642298v2.full
please check Taxonomic classification section.

[larssnip]

Thanks for this! I think I actually have R code for doing what you do in awk in your shell script. But I will for now just let metabuli build without these extra CDS files.

I may come back later, when I want to extend the GTDB with Eukrayotes and Virus. I have done this with kraken2+bracken, and want to use the exact same set of genomes/taxonomy for metabuli. I may come back with more questions then. Thank you for now!

LS

[jaebeom-kim]

Hi!
I built a GTDB R226 database having a reduced database size.
I included all species representative genomes having completeness > 70% and contamination < 5%.
The created DB includes 129,671 species (~90% of total GTDB species).
The database size is 453 GB.
You can download it here

[jaebeom-kim]

I found some memory leaks. I will let you know when it fixed.

[larssnip]

OK, I built a GTDB r226 myself, including all representative genomes. It worked fine. I have used it already.

I also found that I had to increase the memory allocation on the HPC from 128G even if I specified 128G in metabuli. This is, however, no big deal for us since our nodes can handle a lot more than 128G as long as I allocate it.

Thank you for your efforts, and again I may come back later since I want to make database with GTDB prokaryotes and NCBI eukaryotes, and I have some issues with the taxonomy from the taxonkit github site that you link to...