Gazebo Harmonic WSLg memory leak

[UavYuriyS]

When I run a world with meshes in WSL2, I experience memory leak in the gz sim gui process which can be tracked both from inside the VM with htop and from outside with the Task Manager

Environment

• OS Version: WSL Ubuntu-22.04
• Source or binary build?
  • Binary build, Gazebo Sim, version 8.9.0, Harmonic
• If this is a GUI or sensor rendering bug, describe your GPU and rendering system. Otherwise delete this section.
• There are warnings [GUI] [Wrn] [ColladaLoader.cc:2395] Triangle input semantic: 'COLOR' is currently not supported and [GUI] [Wrn] [Ogre2Heightmap.cc:125] Heightmap final sampling should be 2^n

• running on a dual GPU machine (integrated GPU + discrete GPU)
• running on a multi-GPU machine (it has multiple discrete GPUs)
• running on real hardware
• running in virtual machine: WSLg
• running in Docker/Singularity, No, but the issue persists in Doсker too.
• running remotely (e.g. via SSH)
• running in a cloud
• using VirtualGL, XVFB, Xdummy, XVNC or other indirect rendering utilities
• GPU is concurrently used for other tasks
  • desktop acceleration
  • video decoding (i.e. a playing Youtube video)
  • video encoding
  • CUDA/ROCm computations (Tensorflow, Torch, Caffe running)
  • multiple simulators running at the same time
• other...
  • Rendering system info:
    • On Linux, provide the outputs of the following commands:

      LANG=C glxinfo -B | grep -i '\(direct rendering\|opengl\|profile\)'
      direct rendering: Yes
      Preferred profile: core (0x1)
      Max core profile version: 4.2
      Max compat profile version: 4.2
      Max GLES1 profile version: 1.1
      Max GLES[23] profile version: 3.1
      OpenGL vendor string: Microsoft Corporation
      OpenGL renderer string: D3D12 (NVIDIA GeForce RTX 4070 SUPER)
      OpenGL core profile version string: 4.2 (Core Profile) Mesa 23.2.1-1ubuntu3.1~22.04.3
      OpenGL core profile shading language version string: 4.20
      OpenGL core profile context flags: (none)
      OpenGL core profile profile mask: core profile
      OpenGL version string: 4.2 (Compatibility Profile) Mesa 23.2.1-1ubuntu3.1~22.04.3
      OpenGL shading language version string: 4.20
      OpenGL context flags: (none)
      OpenGL profile mask: compatibility profile
      OpenGL ES profile version string: OpenGL ES 3.1 Mesa 23.2.1-1ubuntu3.1~22.04.3
      OpenGL ES profile shading language version string: OpenGL ES GLSL ES 3.10
      
      kotik    3680518  0.0  0.0   4024  2032 pts/4    S+   15:38   0:00 grep --color=auto Xorg

    • On Windows, run dxdiag and report the GPU-related information.
    • On Mac OS, open a terminal and type system_profiler SPDisplaysDataType. Copy the output here.
  • Please, attach the ogre.log or ogre2.log file from ~/.gz/rendering

Description

• Expected behavior: Gazebo works and does not hog memory
• Actual behavior: Gazebo slowly eats all the memory, starts swapping and makes the machine unusable. Setting LIBGL_ALWAYS_SOFTWARE to 1 helps, but it is a no-solution because it disables GPU and makes everything excruciatingly slow

This is the world I'm using to test
drone_scene_repro.txt

Here is my ogre2 log
ogre2.log

And dxdiag to boot
DxDiag.txt

Steps to reproduce

1. Work on a machine with WSL2 and NVIDIA GPU installed
2. Launch the world I provided with LIBGL_ALWAYS_SOFTWARE=0 gz sim drone_scene_repro.txt -v, observe memory leak.
3. Launch the world I provided with LIBGL_ALWAYS_SOFTWARE=1 gz sim drone_scene_repro.txt -v, and observe NO memory leak

Output

[traversaro]

If there the leak is only when using the Nvidia driver, could it be possible that the leak is in the Nvidia driver itself or d3d12?

Perhaps you can try with a newer mesa version to get a newer version of d3d12?

[UavYuriyS]

Perhaps you can try with a newer mesa version to get a newer version of d3d12?

Sure!
I tried to install a newer mesa (25.0.4) from kisak-mesa, but it did not work well with WSLg—the renderer kept defaulting to llvmpipe. So I spun up a fresh Ubuntu-24.04, which has 24.2.8 preinstalled

LANG=C glxinfo -B | grep -i '\(direct rendering\|opengl\|profile\)'
direct rendering: Yes
    Preferred profile: core (0x1)
    Max core profile version: 4.6
    Max compat profile version: 4.6
    Max GLES1 profile version: 1.1
    Max GLES[23] profile version: 3.1
    OpenGL vendor string: Microsoft Corporation
    OpenGL renderer string: D3D12 (NVIDIA GeForce RTX 4070 SUPER)
    OpenGL core profile version string: 4.6 (Core Profile) Mesa 24.2.8-1ubuntu1~24.04.1
    OpenGL core profile shading language version string: 4.60
    OpenGL core profile context flags: (none)
    OpenGL core profile profile mask: core profile
    OpenGL version string: 4.6 (Compatibility Profile) Mesa 24.2.8-1ubuntu1~24.04.1
    OpenGL shading language version string: 4.60
    OpenGL context flags: (none)
    OpenGL profile mask: compatibility profile
    OpenGL ES profile version string: OpenGL ES 3.1 Mesa 24.2.8-1ubuntu1~24.04.1
    OpenGL ES profile shading language version string: OpenGL ES GLSL ES 3.10

I then installed Gazebo Harmonic, and unfortunately, the leak is still there.
I might as well try to install the latest Gazebo on 24.04 and see what happens

UPD: With Gazebo Ionic the problem still persists.

Interestingly, ogre2 log contains mentions of the resource leak on all versions I've tried so far.

11:13:27: Can't assign material scene::Material(65499) because this Material does not exist. Have you forgotten to define it in a .material script?
11:13:27: WARNING: Deleting mapped buffer without having it unmapped. This is often sign of a resource leak or a bad pattern. Umapping the buffer for you...
11:13:27: Texture memory budget exceeded. Stalling GPU.
11:13:27: Stalling was not enough. Freeing memory.

[traversaro]

Thanks for the in-depth analysis. So I guess you also tried with an empty world, and this does not happen?

In your world you have two models:

        <include>
            <uri>
                https://fuel.ignitionrobotics.org/1.0/OpenRobotics/models/Fortress island
            </uri>
        </include>
            
        <include>
            <uri>
                https://fuel.ignitionrobotics.org/1.0/OpenRobotics/models/Fortress heightmap
            </uri>
        </include>

Did you tried to only load one or another, and check if the behavior is still there?

[UavYuriyS]

I did some more digging, removing various parts of the world here and there and that's what I found:

• The effect seems to be cumulative, so both models contribute to the leak but at different rates though.
• The heightmap model which represents mountains around the island contributes a little bit, while the island itself contributes a lot
  • Island is at about 60 MB/s, heightmap is much smaller, around 0.6MB/s but it is there -- if left to its own devices it will eat up ram
  • Numbers are precise up to the order of magnitude, I just eyeballed them from the Task Manager
• Removing various parts of island model -- like chairs, benches and other stuff -- decreases the mem leak rate
• Removing textures greatly reduces the rate (2.65 MB/s with full world) but that's yeah, not the best solution

I'll try with an empty world and upd my comment

UPD: Yup it still leaks but very slowly

[UavYuriyS]

I've also tried to create a world with my own obj files manually; and it also leaks memory, unfortunately

[Knueppl]

Hello,

I encountered the same problem. Being on Windows 11 using WSL2 with Ubuntu 24.04 and ROS Jazzy. I got also an Nvidia GPU. Launching the world Depot results in a memory leak of 5MB/s.

Launching an empty world is fine. No memory leak!

I tried valgrind, but discovered nothing. But I am also not sure that I used it correctly because of the ruby command.

Is there any information/test that would help you to understand the problem?

Greetings