How to increase cuda memory

[wangyue12]

I run the house_double_floor_upper,then get the following error
, FILE C:\g\101116487\physx\source\gpuarticulation\src\PxgArticulationCore.cpp, LINE 677
2025-03-21 11:03:45 [43,654ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2025-03-21 11:03:45 [43,655ms] [Error] [omni.physx.plugin] PhysX error: GPU artiSolveInternalTendonAndMimicJointConstraints1T fail to launch kernel!!
, FILE C:\g\101116487\physx\source\gpuarticulation\src\PxgArticulationCore.cpp, LINE 698
2025-03-21 11:03:45 [43,668ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2025-03-21 11:03:45 [43,669ms] [Error] [omni.physx.plugin] PhysX error: GPU solveInternalConstraints fail to launch kernel!!
, FILE C:\g\101116487\physx\source\gpuarticulation\src\PxgArticulationCore.cpp, LINE 727
2025-03-21 11:03:45 [43,669ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2025-03-21 11:03:45 [43,669ms] [Error] [omni.physx.plugin] PhysX error: GPU compute solver bodies average velocities fail to launch kernel!!
, FILE C:\g\101116487\physx\source\gpusolver\src\PxgTGSCudaSolverCore.cpp, LINE 1896
2025-03-21 11:03:45 [43,669ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2025-03-21 11:03:45 [43,669ms] [Error] [omni.physx.plugin] PhysX error: GPU mWritebackBlockFunction fail to launch kernel!!
, FILE C:\g\101116487\physx\source\gpusolver\src\PxgTGSCudaSolverCore.cpp, LINE 1129
2025-03-21 11:03:45 [43,669ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2025-03-21 11:03:45 [43,669ms] [Error] [omni.physx.plugin] PhysX error: SynchronizeStreams cuEventRecord failed with error 700
, FILE C:\g\101116487\physx\source\gpucommon\include\PxgCudaUtils.h, LINE 79
2025-03-21 11:03:45 [43,669ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2025-03-21 11:03:45 [43,669ms] [Error] [omni.physx.plugin] PhysX error: SynchronizeStreams cuStreamWaitEvent failed with error 700
, FILE C:\g\101116487\physx\source\gpucommon\include\PxgCudaUtils.h, LINE 85
2025-03-21 11:03:45 [43,669ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2025-03-21 11:03:45 [43,669ms] [Error] [omni.physx.plugin] PhysX error: GPU integrateCoreParallel fail to launch kernel!!
, FILE C:\g\101116487\physx\source\gpusolver\src\PxgTGSCudaSolverCore.cpp, LINE 2157
2025-03-21 11:03:45 [43,670ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2025-03-21 11:03:45 [43,670ms] [Error] [omni.physx.plugin] PhysX error: SynchronizeStreams cuEventRecord failed with error 700
, FILE C:\g\101116487\physx\source\gpucommon\include\PxgCudaUtils.h, LINE 79
2025-03-21 11:03:45 [43,670ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2025-03-21 11:03:45 [43,670ms] [Error] [omni.physx.plugin] PhysX error: SynchronizeStreams cuStreamWaitEvent failed with error 700
, FILE C:\g\101116487\physx\source\gpucommon\include\PxgCudaUtils.h, LINE 85
2025-03-21 11:03:45 [43,670ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2025-03-21 11:03:45 [43,670ms] [Error] [omni.physx.plugin] PhysX error: GPU kernel 'markAggregateBoundsUpdated' failed to launch with error 700!!
, FILE C:\g\101116487\physx\source\gpubroadphase\src\PxgAABBManager.cpp, LINE 1350
2025-03-21 11:03:45 [43,670ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2025-03-21 11:03:45 [43,671ms] [Error] [omni.physx.plugin] PhysX error: SynchronizeStreams cuEventRecord failed with error 700
, FILE C:\g\101116487\physx\source\gpucommon\include\PxgCudaUtils.h, LINE 79
2025-03-21 11:03:45 [43,671ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2025-03-21 11:03:45 [43,671ms] [Error] [omni.physx.plugin] PhysX error: SynchronizeStreams cuStreamWaitEvent failed with error 700
, FILE C:\g\101116487\physx\source\gpucommon\include\PxgCudaUtils.h, LINE 85
2025-03-21 11:03:45 [43,671ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2025-03-21 11:03:45 [43,871ms] [Error] [omni.physx.plugin] PhysX error: PhysX Internal CUDA error. Simulation cannot continue! Error code 700!
, FILE C:\g\101116487\physx\source\physx\src\NpScene.cpp, LINE 3059
2025-03-21 11:03:45 [43,874ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2025-03-21 11:03:46 [44,629ms] [Warning] [omni.syntheticdata.plugin] SdRenderVarPtr missing valid input renderVar LdrColorSDhost
2025-03-21 11:03:47 [45,533ms] [Error] [carb.cudainterop.plugin] CUDA error 700: cudaErrorIllegalAddress - an illegal memory access was encountered)
2025-03-21 11:03:47 [45,537ms] [Error] [carb.cudainterop.plugin] Failed to wait on external semaphore in CUDA.
2025-03-21 11:03:47 [45,538ms] [Error] [gpu.foundation.plugin] Wait for external semaphore failed in CUDA submission in command list "Render graph command list (Render queue 0, device 0, frame submission index 0)"
Traceback (most recent call last):
File "C:\Users\kunyue\miniconda3\envs\omnigibson\lib\runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Users\kunyue\miniconda3\envs\omnigibson\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
File "E:\kunyue\2025\research\multimodal\OmniGibson\omnigibson\examples\simulation\capture_scene_graph.py", line 154, in
capture(args.scene_name)
File "E:\kunyue\2025\research\multimodal\OmniGibson\omnigibson\examples\simulation\capture_scene_graph.py", line 75, in capture
env = og.Environment(configs=config)
File "E:\kunyue\2025\research\multimodal\OmniGibson\omnigibson\utils\python_utils.py", line 93, in wrapper
func(*values.args, **values.kwargs)
File "E:\kunyue\2025\research\multimodal\OmniGibson\omnigibson\envs\env_base.py", line 128, in init
self.load()
File "E:\kunyue\2025\research\multimodal\OmniGibson\omnigibson\envs\env_base.py", line 429, in load
self._load_scene()
File "E:\kunyue\2025\research\multimodal\OmniGibson\omnigibson\envs\env_base.py", line 257, in _load_scene
og.sim.import_scene(self._scene)
File "E:\kunyue\2025\research\multimodal\OmniGibson\omnigibson\simulator.py", line 715, in import_scene
scene.initialize()
File "E:\kunyue\2025\research\multimodal\OmniGibson\omnigibson\scenes\scene_base.py", line 497, in initialize
assert og.sim.is_playing(), "Simulator must be playing in order to initialize the scene!"
AssertionError: Simulator must be playing in order to initialize the scene!
2025-03-21 11:03:47 [45,683ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal
2025-03-21 11:03:47 [45,686ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal
2025-03-21 11:03:47 [45,687ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal
2025-03-21 11:03:47 [45,687ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal
2025-03-21 11:03:47 [45,688ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal
2025-03-21 11:03:47 [45,688ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal
2025-03-21 11:03:47 [45,689ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal
2025-03-21 11:03:47 [45,689ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal
2025-03-21 11:03:47 [45,690ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal
2025-03-21 11:03:47 [45,690ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal
2025-03-21 11:03:47 [45,690ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal
2025-03-21 11:03:47 [45,691ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Core' for removal
2025-03-21 11:03:47 [45,698ms] [Warning] [omni.physx.plugin] PhysX warning: C:\g\101116487\physx\source\gpucommon\src\PxgCudaMemoryAllocator.cpp, FILE C:\g\101116487\physx\source\gpucommon\src\PxgCudaMemoryAllocator.cpp, LINE 167
2025-03-21 11:03:47 [45,698ms] [Warning] [omni.physx.plugin] PhysX warning: C:\g\101116487\physx\source\gpucommon\src\PxgCudaMemoryAllocator.cpp, FILE C:\g\101116487\physx\source\gpucommon\src\PxgCudaMemoryAllocator.cpp, LINE 167
2025-03-21 11:03:47 [45,698ms] [Warning] [omni.physx.plugin] PhysX warning: C:\g\101116487\physx\source\gpucommon\src\PxgCudaMemoryAllocator.cpp, FILE C:\g\101116487\physx\source\gpucommon\src\PxgCudaMemoryAllocator.cpp, LINE 167
2025-03-21 11:03:47 [45,699ms] [Warning] [omni.physx.plugin] PhysX warning: C:\g\101116487\physx\source\gpucommon\src\PxgCudaMemoryAllocator.cpp, FILE C:\g\101116487\physx\source\gpucommon\src\PxgCudaMemoryAllocator.cpp, LINE 167
2025-03-21 11:03:47 [45,699ms] [Warning] [omni.physx.plugin] PhysX warning: C:\g\101116487\physx\source\gpucommon\src\PxgCudaMemoryAllocator.cpp, FILE C:\g\101116487\physx\source\gpucommon\src\PxgCudaMemoryAllocator.cpp, LINE 167
2025-03-21 11:03:47 [45,700ms] [Warning] [omni.physx.plugin] PhysX warning: C:\g\101116487\physx\source\gpucommon\src\PxgCudaMemoryAllocator.cpp, FILE C:\g\101116487\physx\source\gpucommon\src\PxgCudaMemoryAllocator.cpp, LINE 167
2025-03-21 11:03:47 [45,700ms] [Warning] [omni.physx.plugin] PhysX warning: C:\g\101116487\physx\source\gpucommon\src\PxgCudaMemoryAllocator.cpp, FILE C:\g\101116487\physx\source\gpucommon\src\PxgCudaMemoryAllocator.cpp, LINE 167
2025-03-21 11:03:47 [45,705ms] [Warning] [omni.physx.plugin] PhysX warning: C:\g\101116487\physx\source\gpucommon\src\PxgCudaMemoryAllocator.cpp, FILE C:\g\101116487\physx\source\gpucommon\src\PxgCudaMemoryAllocator.cpp, LINE 167
2025-03-21 11:03:47 [45,706ms] [Warning] [omni.physx.plugin] PhysX warning: C:\g\101116487\physx\source\gpucommon\src\PxgCudaMemoryAllocator.cpp, FILE C:\g\101116487\physx\source\gpucommon\src\PxgCudaMemoryAllocator.cpp, LINE 167
2025-03-21 11:03:47 [45,706ms] [Warning] [omni.physx.plugin] PhysX warning: C:\g\101116487\physx\source\gpucommon\src\PxgCudaMemoryAllocator.cpp, FILE C:\g\101116487\physx\source\gpucommon\src\PxgCudaMemoryAllocator.cpp, LINE 167
2025-03-21 11:03:47 [45,709ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 700., FILE C:\g\101116487\physx\source\cudamanager\src\CudaContextManager.cpp, LINE 817
2025-03-21 11:03:47 [45,709ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 700., FILE C:\g\101116487\physx\source\cudamanager\src\CudaContextManager.cpp, LINE 817
2025-03-21 11:03:47 [45,709ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 700., FILE C:\g\101116487\physx\source\cudamanager\src\CudaContextManager.cpp, LINE 81