Python script error: Too many open files in system

[Nataliia5722]

Hi Team!
We started experiencing strange error for argo-workflows:

Could not import runpy module
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/runpy.py", line 15, in <module>
  File "/usr/local/lib/python3.9/importlib/util.py", line 2, in <module>
  File "/usr/local/lib/python3.9/importlib/abc.py", line 17, in <module>
  File "/usr/local/lib/python3.9/typing.py", line 21, in <module>
  File "/usr/local/lib/python3.9/collections/__init__.py", line 30, in <module>
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 846, in exec_module
  File "<frozen importlib._bootstrap_external>", line 982, in get_code
  File "<frozen importlib._bootstrap_external>", line 1039, in get_data
OSError: [Errno 23] Too many open files in system: '/usr/local/lib/python3.9/heapq.py'
/argo/staging/script: cannot make pipe for command substitution: Too many open files in system

From the configuration side nothing has changed, and we have workflows that were running for around 6 months successfully and then at one moment started failing with OSError: [Errno 23] Too many open files in system

Could you, please, advise how it can be fixed?

[Joibel]

This is not argo-workflows fault.

You are running python inside your workflow pods, which are just normal kubernetes pods.
Some thoughts:

• you are now processing more files than you used to, so are running out of file handles
• your nodes are busier than they used to be, so the nodes are running out of file handles
• your code has been updated and now doesn't close a file handle that it should
• your kubernetes cluster has been updated and the nodes are more constrained than they used to be.

The argo-workflows community can't help fix this for you

[Nataliia5722]

Thank you!

That helps a lot!