Maps generator fails with SIGABRT

[RicoElectrico]

Hi, I just could not get the maps generator to work and would appreciate any help. While the binary standalone reports its version (so it's probably not corrupted):

haxor@brzozows-5501:~/src/omim-build-release$ ./generator_tool --version
generator_tool version 2023.05.23-1-3ba272e1-Linux

... the map generator fails.

haxor@brzozows-5501:~/src/organicmaps/tools/python$ python3 -m maps_generator --countries="Poland_Pomeranian Voivodeship" --skip="Coastline"
[2023-05-26 00:09:49,563] INFO env Start setup ...
[2023-05-26 00:09:49,563] INFO env Check osm tools ...
[2023-05-26 00:09:49,564] INFO env Osm tools found - /home/haxor/osmctools/osmconvert, /home/haxor/osmctools/osmupdate, /home/haxor/osmctools/osmfilter
[2023-05-26 00:09:49,564] INFO env Check generator tool ...
[2023-05-26 00:09:49,637] INFO env Looking for generator tool in /home/haxor/src/omim-build-release ...
[2023-05-26 00:09:49,637] INFO env Generator tool found - /home/haxor/src/omim-build-release/generator_tool
[2023-05-26 00:09:49,637] INFO env Build name is 2023_05_26__00_09_49.
[2023-05-26 00:09:49,637] INFO env Build path is /home/haxor/maps_build/2023_05_26__00_09_49.
[2023-05-26 00:09:49,637] INFO env Create /home/haxor/maps_build/2023_05_26__00_09_49 ...
[2023-05-26 00:09:49,638] INFO env Create /home/haxor/maps_build/2023_05_26__00_09_49/intermediate_data ...
[2023-05-26 00:09:49,638] INFO env Create /home/haxor/maps_build/2023_05_26__00_09_49/intermediate_data/borders ...
[2023-05-26 00:09:49,638] INFO env Create /home/haxor/maps_build/2023_05_26__00_09_49/draft ...
[2023-05-26 00:09:49,639] INFO env Create /home/haxor/maps_build/2023_05_26__00_09_49/osm2ft ...
[2023-05-26 00:09:49,639] INFO env The following 1 maps will build: Poland_Pomeranian Voivodeship.
[2023-05-26 00:09:49,639] INFO env Finish setup
[2023-05-26 00:09:49,639] INFO env Create /home/haxor/maps_build/2023_05_26__00_09_49/logs ...
[2023-05-26 00:09:49,640] INFO stages Stage DownloadAndConvertPlanet: start ...
[2023-05-26 00:09:49,640] INFO env Create /home/haxor/maps_build/2023_05_26__00_09_49/status ...
[2023-05-26 00:09:49,647] INFO file Trying to download /home/haxor/maps_build/2023_05_26__00_09_49/planet.osm.pbf from http://download.geofabrik.de/europe/poland/pomorskie-latest.osm.pbf.
[2023-05-26 00:09:49,647] INFO file Trying to download /home/haxor/maps_build/2023_05_26__00_09_49/planet.osm.pbf.md5 from http://download.geofabrik.de/europe/poland/pomorskie-latest.osm.pbf.md5.
[2023-05-26 00:09:49,800] INFO file File /home/haxor/maps_build/2023_05_26__00_09_49/planet.osm.pbf.md5 was downloaded from http://download.geofabrik.de/europe/poland/pomorskie-latest.osm.pbf.md5.
[2023-05-26 00:09:54,726] INFO file File /home/haxor/maps_build/2023_05_26__00_09_49/planet.osm.pbf was downloaded from http://download.geofabrik.de/europe/poland/pomorskie-latest.osm.pbf.
[2023-05-26 00:09:57,249] INFO stages Stage DownloadAndConvertPlanet: finished in 0:00:07.609380
[2023-05-26 00:09:57,249] INFO stages Stage Preprocess: start ...
[2023-05-26 00:09:57,256] INFO gen_tool Run generator tool [generator_tool version 2023.05.23-1-3ba272e1-Linux]: /home/haxor/src/omim-build-release/generator_tool --threads_count=1 --intermediate_data_path=/home/haxor/maps_build/2023_05_26__00_09_49/intermediate_data --cache_path=/home/haxor/maps_build/2023_05_26__00_09_49/intermediate_data --osm_file_type=o5m --osm_file_name=/home/haxor/maps_build/2023_05_26__00_09_49/planet.o5m --node_storage=map --user_resource_path=/home/haxor/src/organicmaps/data --preprocess=true
[2023-05-26 00:10:00,496] INFO stages Stage Preprocess: finished in 0:00:03.246469
[2023-05-26 00:10:00,496] INFO stages Stage Features: start ...
[2023-05-26 00:10:00,497] INFO stages Test test_booking_data was skipped.
[2023-05-26 00:10:00,506] INFO gen_tool Run generator tool [generator_tool version 2023.05.23-1-3ba272e1-Linux]: /home/haxor/src/omim-build-release/generator_tool --threads_count=12 --data_path=/home/haxor/maps_build/2023_05_26__00_09_49/intermediate_data --intermediate_data_path=/home/haxor/maps_build/2023_05_26__00_09_49/intermediate_data --cache_path=/home/haxor/maps_build/2023_05_26__00_09_49/intermediate_data --osm_file_type=o5m --osm_file_name=/home/haxor/maps_build/2023_05_26__00_09_49/planet.o5m --node_storage=map --user_resource_path=/home/haxor/src/organicmaps/data --dump_cities_boundaries=true --cities_boundaries_data=/home/haxor/maps_build/2023_05_26__00_09_49/intermediate_data/cities_boundaries.bin --generate_features=true --isolines_path= --generate_packed_borders=true
[2023-05-26 00:11:18,425] INFO stages Stage Features: finished in 0:01:17.928737
[2023-05-26 00:11:18,426] ERROR env Uncaught exception: The launch of /home/haxor/src/omim-build-release/generator_tool --threads_count=12 --data_path=/home/haxor/maps_build/2023_05_26__00_09_49/intermediate_data --intermediate_data_path=/home/haxor/maps_build/2023_05_26__00_09_49/intermediate_data --cache_path=/home/haxor/maps_build/2023_05_26__00_09_49/intermediate_data --osm_file_type=o5m --osm_file_name=/home/haxor/maps_build/2023_05_26__00_09_49/planet.o5m --node_storage=map --user_resource_path=/home/haxor/src/organicmaps/data --dump_cities_boundaries=true --cities_boundaries_data=/home/haxor/maps_build/2023_05_26__00_09_49/intermediate_data/cities_boundaries.bin --generate_features=true --isolines_path= --generate_packed_borders=true failed.
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/haxor/src/organicmaps/tools/python/maps_generator/__main__.py", line 303, in <module>
    main()
  File "/home/haxor/src/organicmaps/tools/python/maps_generator/__main__.py", line 299, in main
    generate_maps(env, from_stage)
  File "/home/haxor/src/organicmaps/tools/python/maps_generator/maps_generator.py", line 43, in generate_maps
    run_generation(env, stages, from_stage)
  File "/home/haxor/src/organicmaps/tools/python/maps_generator/maps_generator.py", line 24, in run_generation
    generation.run(from_stage)
  File "/home/haxor/src/organicmaps/tools/python/maps_generator/generator/generation.py", line 79, in run
    self.run_stages()
  File "/home/haxor/src/organicmaps/tools/python/maps_generator/generator/generation.py", line 85, in run_stages
    stage(self.env)
  File "/home/haxor/src/organicmaps/tools/python/maps_generator/generator/stages.py", line 87, in __call__
    return self.apply(env, **self.args)
  File "/home/haxor/src/organicmaps/tools/python/maps_generator/generator/stages.py", line 199, in apply
    method(obj, env, *args, **kwargs)
  File "/home/haxor/src/organicmaps/tools/python/maps_generator/generator/stages.py", line 371, in apply
    method(obj, env, *args, **kwargs)
  File "/home/haxor/src/organicmaps/tools/python/maps_generator/generator/stages.py", line 295, in apply
    method(obj, env, *args, **kwargs)
  File "/home/haxor/src/organicmaps/tools/python/maps_generator/generator/stages_declaration.py", line 136, in apply
    steps.step_features(env, **extra)
  File "/home/haxor/src/organicmaps/tools/python/maps_generator/generator/steps.py", line 118, in step_features
    run_gen_tool(
  File "/home/haxor/src/organicmaps/tools/python/maps_generator/generator/gen_tool.py", line 162, in run_gen_tool
    GenTool(*args, **kwargs).run()
  File "/home/haxor/src/organicmaps/tools/python/maps_generator/generator/gen_tool.py", line 138, in run
    wait_and_raise_if_fail(self)
  File "/home/haxor/src/organicmaps/tools/python/maps_generator/generator/exceptions.py", line 38, in wait_and_raise_if_fail
    raise BadExitStatusError(f"The launch of {' '.join(p.args)} failed.")
maps_generator.generator.exceptions.BadExitStatusError: The launch of /home/haxor/src/omim-build-release/generator_tool --threads_count=12 --data_path=/home/haxor/maps_build/2023_05_26__00_09_49/intermediate_data --intermediate_data_path=/home/haxor/maps_build/2023_05_26__00_09_49/intermediate_data --cache_path=/home/haxor/maps_build/2023_05_26__00_09_49/intermediate_data --osm_file_type=o5m --osm_file_name=/home/haxor/maps_build/2023_05_26__00_09_49/planet.o5m --node_storage=map --user_resource_path=/home/haxor/src/organicmaps/data --dump_cities_boundaries=true --cities_boundaries_data=/home/haxor/maps_build/2023_05_26__00_09_49/intermediate_data/cities_boundaries.bin --generate_features=true --isolines_path= --generate_packed_borders=true failed.

Dmesg log:

[ 2103.385174] potentially unexpected fatal signal 6.
[ 2103.385184] CPU: 1 PID: 2190 Comm: generator_tool Not tainted 5.15.90.1-microsoft-standard-WSL2 #1
[ 2103.385192] RIP: 0033:0x7f1208668a7c
[ 2103.385203] Code: 41 89 c5 41 f7 dd eb 80 66 0f 1f 44 00 00 b8 ba 00 00 00 0f 05 89 c5 e8 42 56 05 00 44 89 e2 89 ee 89 c7 b8 ea 00 00 00 0f 05 <41> 89 c5 41 f7 dd 3d 00 f0 ff ff b8 00 00 00 00 44 0f 46 e8 e9 6d
[ 2103.385208] RSP: 002b:00007f1201d00db0 EFLAGS: 00000246
[ 2103.385213] RAX: 0000000000000000 RBX: 00007f1201d02640 RCX: 00007f1208668a7c
[ 2103.385217] RDX: 0000000000000006 RSI: 0000000000000770 RDI: 000000000000074c
[ 2103.385220] RBP: 0000000000000770 R08: 00007f11c44cacf0 R09: 00007f12085d3b8c
[ 2103.385223] R10: 00007f1208659400 R11: 0000000000000246 R12: 0000000000000006
[ 2103.385226] R13: 0000000000000016 R14: 00007f11c452e460 R15: 00007f1201d01578
[ 2103.385229] FS:  00007f1201d02640 GS:  0000000000000000

Ini file:

[Main]
# If the flag DEBUG is set a special small planet file will be downloaded.
DEBUG: 0
# The path where the planet will be downloaded and the maps are generated.
MAIN_OUT_PATH: ~/maps_build
# The path where caches for nodes, ways, relations are stored.
# CACHE_PATH:


[Developer]
# The path where the generator_tool will be searched.
BUILD_PATH: ~/src/omim-build-release
# The path to the project directory omim.
OMIM_PATH: ~/src/organicmaps


[Generator tool]
# The path to the omim/data.
USER_RESOURCE_PATH: ${Developer:OMIM_PATH}/data
# Do not change it. This is determined automatically.
# NODE_STORAGE: map


[Osm tools]
# The path to the osmctools sources.
OSM_TOOLS_SRC_PATH: ${Developer:OMIM_PATH}/tools/osmctools
# The path where osmctools will be searched or will be built.
OSM_TOOLS_PATH: ~/osmctools


[Stages]
# Run osmupdate tool for planet.
NEED_PLANET_UPDATE: 0
# Auto detection.
THREADS_COUNT_FEATURES_STAGE: 1
# If you want to calculate diffs, you need to specify, where old maps are
DATA_ARCHIVE_DIR: ${Generator tool:USER_RESOURCE_PATH}
# You may specify, how many versions in the archive to use for diff calculation
DIFF_VERSION_DEPTH: 2


[Logging]
# The path where maps_generator log will be saved.
# LOG_FILE_PATH: generation.log


[External]
# Note: If you want to set a directory name you have to add "/" to the end of url.
# In each field where you need to specify a URL, you can specify the path to the file system using file:///path/to/file

# The url to the planet file.
 PLANET_URL: http://download.geofabrik.de/europe/poland/pomorskie-latest.osm.pbf
# The url to the file with md5 sum of the planet.
 PLANET_MD5_URL: http://download.geofabrik.de/europe/poland/pomorskie-latest.osm.pbf.md5
# The base url to WorldCoasts.geom and WorldCoasts.rawgeom (without file name).
# Files latest_coasts.geom and latest_coasts.rawgeom must be at this URL.
# For example, if PLANET_COASTS_URL = https://somesite.com/download/
# The https://somesite.com/download/latest_coasts.geom url will be used to download latest_coasts.geom and
# the https://somesite.com/download/latest_coasts.rawgeom url will be used to download latest_coasts.rawgeom.
# PLANET_COASTS_URL:
# Set to 'true' to build special routing section in World.mwm for alerting about absent regions without which the
# route can't be built.
NEED_BUILD_WORLD_ROADS: true
# The url to the subway file.
SUBWAY_URL: https://cdn.organicmaps.app/subway.json

# The url of the location with the transit files extracted from GTFS.
# TRANSIT_URL:

# Urls for production maps generation.
# UGC_URL:
# HOTELS_URL:
# PROMO_CATALOG_CITIES:
# POPULARITY_URL:
# FOOD_URL:
# FOOD_TRANSLATIONS_URL:
# SRTM_PATH:
# ISOLINES_PATH:
# UK_POSTCODES_URL:
# US_POSTCODES_URL:


[Common]
# Auto detection.
THREADS_COUNT: 1


[Stats]
# Path to rules for calculating statistics by type
STATS_TYPES_CONFIG: ${Developer:OMIM_PATH}/tools/python/maps_generator/var/etc/stats_types_config.txt

[biodranik]

You are likely the first one who tries to run it in WSL...

[RicoElectrico]

Probably ;)
Though WSL2 is mostly a "real" Linux, unlike WSL which has slow I/O and is overall not good. What does differ from vanilla Ubuntu is mostly how systemd and services work, didn't fully figure that out, albeit it should not affect the maps generator
I will do the same steps on a non-WSL2 install and see what happens.

[RicoElectrico]

Tried with Ubuntu 20.04 on a real machine and it works.
Now it leaves the question why it fails on WSL2.

[danvyr]

I have a docker image with generator tool danvyr/organicmap
example how to use it https://github.com/danvyr/osmmapsconverter/blob/master/organicmap/run.sh

And for Poland I do everyday generation https://download.openstreetmap.by/organicmap/

[RicoElectrico]

Thanks, that's a great resource! Although my goal was to experiment with adding new OSM features/tags or getting better isoline sources, which is why I needed to compile it in the first place ;)

[danvyr]

you can try to use my dockerfile for your purpueses https://github.com/danvyr/osmmapsconverter/blob/master/organicmap/Dockerfile

[RicoElectrico]

Woah. I tried to get a core dump by enabling them (ulimit -c unlimited) and it somehow works now. Kind of awkward situation, isn't it 😅