PID files: use file locks

Currently there are various problems with the PID file approach for process management. We can improve the situation easily by using file locks on the PID files. Solves problems and makes it easier to find out if the program is still running. File is locked -> bot is running. no file lock -> bot not running.

However, that means that the file needs to be locked by the bot itself, not by intelmqctl (which is currently the case)!

Not sure if this change is safe and easy enough to be implemented in a maintenance release.

See intelmq.bin.intelmqctl.IntelMQProcessManager.

[gethvi]

@wagner-certat Would it be possible to move the PID file creation from IntelMQProcessManager to the lib.bot? When a bot spawns, it creates it's own PID file. When any fatal exception happens and the bot is about to exit it would delete it's PID file.
The only thing IntelMQProcessManager would do with PID files is that it would check that a process with such PID is running and is in fact the correct process (by checking it's cmdline). If it is not, IntelMQProcessManager would delete the PID file. This would cover cases like SIGKILL to the bot when it wouldn't delete the PID file.

@wagner-certat Would it be possible to move the PID file creation from IntelMQProcessManager to the lib.bot?

Yes, I think this is also necessary if we go for file locks.

When a bot spawns, it creates it's own PID file. When any fatal exception happens and the bot is about to exit it would delete it's PID file.
The only thing IntelMQProcessManager would do with PID files is that it would check that a process with such PID is running and is in fact the correct process (by checking it's cmdline). If it is not, IntelMQProcessManager would delete the PID file. This would cover cases like SIGKILL to the bot when it wouldn't delete the PID file.

Maybe we can even get rid of the command line check then and just check if the file is locked by a processes. I think it's fine to assume that if the PID file is locked, the locking process is the intelmq bot itself. (If some other programs is locking our PID files, that's force majeure :D)