5 Optimizing sensitivity and resource usage

This section outlines how to manage Vclust's run time, memory, and disk space usage while maintaining high sensitivity. These considerations are essential if your dataset contains millions of sequences or has a high level of redundancy.

5.1. Prefilter

The prefilter command generally demands more computational resources (i.e., memory, runtime, and disk space) compared to the align and cluster commands. This is primarily due to its all-versus-all pairwise comparisons of genomes. If not configured correctly, the prefilter step can consume a substantial amount of resources, particularly when dealing with large or highly redundant datasets. In addition, the prefilter command can significantly impact the runtime and memory usage of subsequent align and cluster commands, as it dictates the number of pairwise alignments to be performed. To optimize performance and mitigate excessive resource consumption, it is essential to adjust the following three parameters.

5.1.1. Batch size

Vclust can reduce memory usage by processing genome datasets in smaller, equally sized batches. Although this may slightly increase runtime, it significantly reduces memory requirements without impacting sensitivity. For example, processing 15.5 million IMG/VR contigs in batches of 2 million sequences reduced memory usage from 1 TB to 256 GB, with only a 30-minute increase in runtime.

The --batch-size option specifies the number of sequences to process in each batch.

# Process genomes in batches of 2 million sequences.
./vclust.py prefilter -i genomes.fna -o fltr.txt --batch-size 2000000

5.1.2. Fraction of k-mers

By default, the prefilter command analyzes all k-mers for each genome, but you can limit the analysis to a fraction of k-mers to reduce memory usage and runtime. Reducing the k-mers fraction has minimal impact on sensitivity, as results from a subset of k-mers are generally comparable to the full set. For example, analyzing 20% of the k-mers from 15.5 million IMG/VR contigs recalled nearly all genome pairs with ANI ≥ 95% and AF ≥ 85% (~196 million pairs), with fewer than 100 missed pairs and false positives, while reducing memory and runtime by nearly five-fold. This option does not affect the precision of ANI values in the subsequent align command, as alignments are conducted on full sequences.

The --kmers-fraction option controls the proportion [0-1] of k-mers used in comparisons:

# Process genomes in batches and analyze 20% of k-mers in each genome sequence.
./vclust.py prefilter -i genomes.fna -o fltr.txt --batch-size 2000000 --kmers-fraction 0.2

5.1.3. Maximum number of sequences per query

By default, the prefilter command returns all genome pairs that meet the user-defined thresholds for the minimum number of common k-mers (--min-kmers) and sequence identity (--min-ident). However, for highly redundant datasets (e.g., hundreds of thousands of nearly identical genome sequences), the prefilter step may still pass a large number of genome pairs, increasing both memory usage and runtime. The --max-seqs option limits the number of target sequences reported for each query genome, reducing the overall number of genome pairs passed to alignment. For each query, --max-seqs returns up to n sequences that have passed the --min-kmers and --min-ident filters, and have the highest sequence identity to query sequence. For example, in a dataset containing 1 million nearly identical genomes, the total number of possible genome pairs is almost 500 billion. Setting --max-seqs 1000 reduces this to 1 billion genome pairs, significantly decreasing memory usage and runtime.

# Limit the number of target sequences to 1000 per query genome.
./vclust.py prefilter -i genomes.fna -o fltr.txt --batch-size 100000 --kmers-fraction 0.2 \
--max-seqs 1000