"database is locked" errors

[elmjag]

At HZB, when the app is running a lot of jobs, following exception below appear in the log. Seems to be happening when jobs are finished and possible queued jobs are started.

This is probably happening when jobsd tries to write 'job done' and/or 'job started' to the project database. Possibly cause by some sort of write contention on sqlite level.

May 04 11:11:49 hkl9.psf.bessy.de runappd.py[1557747]: Traceback (most recent call last):
May 04 11:11:49 hkl9.psf.bessy.de runappd.py[1557747]: File "/fraghome/fragadm/miniconda3/envs/FragMAX/lib/python3.10/site-packages/pony/orm/dbapiprovider.py", line 55, in wrap_dbapi_exceptions
May 04 11:11:49 hkl9.psf.bessy.de runappd.py[1557747]: try: return func(provider, *args, **kwargs)
May 04 11:11:49 hkl9.psf.bessy.de runappd.py[1557747]: File "/fraghome/fragadm/miniconda3/envs/FragMAX/lib/python3.10/site-packages/pony/orm/dbapiprovider.py", line 279, in execute
May 04 11:11:49 hkl9.psf.bessy.de runappd.py[1557747]: if arguments is None: cursor.execute(sql)
May 04 11:11:49 hkl9.psf.bessy.de runappd.py[1557747]: sqlite3.OperationalError: database is locked
May 04 11:11:49 hkl9.psf.bessy.de runappd.py[1557747]: During handling of the above exception, another exception occurred:
May 04 11:11:49 hkl9.psf.bessy.de runappd.py[1557747]: Traceback (most recent call last):
May 04 11:11:49 hkl9.psf.bessy.de runappd.py[1557747]: File "/data/fragmaxbin/fragmax/jobs/jobsd/init.py", line 195, in start_jobs_set
May 04 11:11:49 hkl9.psf.bessy.de runappd.py[1557747]: await run_jobs_roots(root_nodes, jobs_table)
May 04 11:11:49 hkl9.psf.bessy.de runappd.py[1557747]: File "/data/fragmaxbin/fragmax/jobs/jobsd/init.py", line 180, in run_jobs_roots
May 04 11:11:49 hkl9.psf.bessy.de runappd.py[1557747]: await asyncio.gather(*co)
May 04 11:11:49 hkl9.psf.bessy.de runappd.py[1557747]: File "/data/fragmaxbin/fragmax/jobs/jobsd/init.py", line 174, in run_jobs_tree
May 04 11:11:49 hkl9.psf.bessy.de runappd.py[1557747]: jobs_table.job_finished(root)
May 04 11:11:49 hkl9.psf.bessy.de runappd.py[1557747]: File "/data/fragmaxbin/fragmax/jobs/jobsd/init.py", line 158, in job_finished
May 04 11:11:49 hkl9.psf.bessy.de runappd.py[1557747]: mark_job_finished(job_node)
May 04 11:11:49 hkl9.psf.bessy.de runappd.py[1557747]: File "", line 2, in mark_job_finished
May 04 11:11:49 hkl9.psf.bessy.de runappd.py[1557747]: File "/fraghome/fragadm/miniconda3/envs/FragMAX/lib/python3.10/site-packages/pony/orm/core.py", line 519, in new_func
May 04 11:11:49 hkl9.psf.bessy.de runappd.py[1557747]: result = func(*args, **kwargs)
May 04 11:11:49 hkl9.psf.bessy.de runappd.py[1557747]: File "/data/fragmaxbin/fragmax/jobs/jobsd/nodes.py", line 50, in mark_job_finished
May 04 11:11:49 hkl9.psf.bessy.de runappd.py[1557747]: job = _get_db_job(job_node)
May 04 11:11:49 hkl9.psf.bessy.de runappd.py[1557747]: File "/data/fragmaxbin/fragmax/jobs/jobsd/nodes.py", line 38, in _get_db_job
May 04 11:11:49 hkl9.psf.bessy.de runappd.py[1557747]: db = _get_db(job_node.project_id)
May 04 11:11:49 hkl9.psf.bessy.de runappd.py[1557747]: File "/data/fragmaxbin/fragmax/jobs/jobsd/nodes.py", line 34, in _get_db
May 04 11:11:49 hkl9.psf.bessy.de runappd.py[1557747]: return get_project_db(conf.PROJECTS_DB_DIR, project_id)
May 04 11:11:49 hkl9.psf.bessy.de runappd.py[1557747]: File "/data/fragmaxbin/fragmax/projects/database.py", line 59, in get_project_db
May 04 11:11:49 hkl9.psf.bessy.de runappd.py[1557747]: return _bind(db_file)
May 04 11:11:49 hkl9.psf.bessy.de runappd.py[1557747]: File "/data/fragmaxbin/fragmax/projects/database.py", line 24, in _bind
May 04 11:11:49 hkl9.psf.bessy.de runappd.py[1557747]: db.generate_mapping(create_tables=create_tables)
May 04 11:11:49 hkl9.psf.bessy.de runappd.py[1557747]: File "/fraghome/fragadm/miniconda3/envs/FragMAX/lib/python3.10/site-packages/pony/orm/core.py", line 1138, in generate_mapping
May 04 11:11:49 hkl9.psf.bessy.de runappd.py[1557747]: elif check_tables: database.check_tables()
May 04 11:11:49 hkl9.psf.bessy.de runappd.py[1557747]: File "", line 2, in check_tables
May 04 11:11:49 hkl9.psf.bessy.de runappd.py[1557747]: File "/fraghome/fragadm/miniconda3/envs/FragMAX/lib/python3.10/site-packages/pony/orm/core.py", line 506, in new_func
May 04 11:11:49 hkl9.psf.bessy.de runappd.py[1557747]: return func(*args, **kwargs)
May 04 11:11:49 hkl9.psf.bessy.de runappd.py[1557747]: File "/fraghome/fragadm/miniconda3/envs/FragMAX/lib/python3.10/site-packages/pony/orm/core.py", line 1211, in check_tables
May 04 11:11:49 hkl9.psf.bessy.de runappd.py[1557747]: database.schema.check_tables(database.provider, connection)
May 04 11:11:49 hkl9.psf.bessy.de runappd.py[1557747]: File "/fraghome/fragadm/miniconda3/envs/FragMAX/lib/python3.10/site-packages/pony/orm/dbschema.py", line 78, in check_tables
May 04 11:11:49 hkl9.psf.bessy.de runappd.py[1557747]: provider.execute(cursor, sql)
May 04 11:11:49 hkl9.psf.bessy.de runappd.py[1557747]: File "", line 2, in execute
May 04 11:11:49 hkl9.psf.bessy.de runappd.py[1557747]: File "/fraghome/fragadm/miniconda3/envs/FragMAX/lib/python3.10/site-packages/pony/orm/dbapiprovider.py", line 72, in wrap_dbapi_exceptions
May 04 11:11:49 hkl9.psf.bessy.de runappd.py[1557747]: raise OperationalError(e)
May 04 11:11:49 hkl9.psf.bessy.de runappd.py[1557747]: pony.orm.dbapiprovider.OperationalError: database is locked