Any strategy for speeding up cogeo_mosaic.mosaic.MosaicJSON.from_urls requests?

[prncevince]

There is a max_threads argument in MosaicJSON.from_urls. It's set to a default value of 20.

cogeo-mosaic/cogeo_mosaic/mosaic.py

Lines 240 to 248 in 4f88749

	def from_urls(
	cls,
	urls: Sequence[str],
	minzoom: Optional[int] = None,
	maxzoom: Optional[int] = None,
	max_threads: int = 20,
	quiet: bool = True,
	**kwargs,
	):

Ultimately, this gets passed to get_footprints

cogeo-mosaic/cogeo_mosaic/utils.py

Lines 79 to 84 in 4f88749

	def get_footprints(
	dataset_list: Sequence[str],
	tms: morecantile.TileMatrixSet = WEB_MERCATOR_TMS,
	max_threads: int = 20,
	quiet: bool = True,
	) -> List:

... and then to a concurrent.futures.ThreadPoolExecutor call.

cogeo-mosaic/cogeo_mosaic/utils.py

Line 105 in 4f88749

with futures.ThreadPoolExecutor(max_workers=max_threads) as executor:

I am no concurrent.futures python GIL expert. On my MBP M1 Pro with 32GB of RAM, import os;os.cpu_count() reports 10, which tells me that I have 10 CPU cores.

I still find that MosaicJSON.from_urls runs slow with the default max_threads=20. Are there any strategies for speeding things up? Thank you!

[prncevince]

I tried setting max_threads=10000, but this basically has the same performance.

I'm creating mosaics of up to ~9,000 COG tiles, so I thought that setting max_threads around the number of tiles could help run all of the requests on the COG URLs concurrently using lots of threads to speed up the concurrent.futures part of from_urls. However, I'm guessing there's more to it to actually do this.

I'm running python 3.11.5 & cogeo-mosiac 7.0.0

[vincentsarago]

Hi @prncevince
Sadly there is no magic, we using multi thread to speed things up when it can, but ultimately this will have advantages only when dealing with I/O while the get_footprints on local files will be mostly a CPU process (GDAL).

if you are dealing with a lot of files you may try:

• use multi-processing to get the data footprint

 cat list_of_cog.tx | parallel -j 10 rio bounds {} >> features.geojson

• create mosaic from the list of features

cogeo-mosaic create-from-features features.geojson --minzoom 10 --maxzoom 12 --property "title" -o mosaic.json

☝️ you'll need to provide the minzoom and maxzoom which will depends on your COG assets (you can find that using rio-cogeo CLI: rio cogeo info )