Provide progress updates when installing/removing packages (opkg)

[chotea]

We need to define a way to provide progress updates when installing/removing packages. Before creating a PR I want to make sure that we all approve the solution for this since it can be applied to any pkg implementation.

In the following lines I will refer to opkg and any solution we chose should not be the default case (we will execute the "progress updates code path" only if a certain key-value pair is present in kwargs).

First task to solve is how we can get real-time information from opkg during an opkg install operation. From what I know opkg doesn't support writing the output to a specific file.
One way to do this is by reading the stdout through pipe. Something like this:

proc = subprocess.Popen(cmd, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
while True:
try:
output = proc.stdout.readline().decode("utf-8")
except:
...
if output == '' and proc.poll() is not None:
break
...

So we have the updates from opkg but what can we do to inform the master?

1. We could fire an event to salt-master bus from here(opkg.install function) to inform the master which package is currently being installed/downloaded. We can use a time interval that can be provided through kwargs to avoid the congestion of salt-master event bus. In addition to that we could count the number of packages that were installed and provide this number as well.

2. We can write the output to a file (/var/volatile/something) and let another salt component be responsible of notifying the master with the updates. My first thought is that the file should have the same name as the id of the job and it should be deleted after the command is completed (here we could have problems if we run two pkg.install jobs in the same time, I will think about it if we go with this solution). I feel that the update load should contain the job_id to be as explicit as possible. Then we could have a beacon that can read the file and fire the event with the updates to the salt master. The beacon will do nothing if it doesn't find a file in that specific directory. Or we could have an inotify beacon that is watching the folder for updates and implement a module that can process the file content and return the progress.

Do we cross the red line if we implement the first solution? Is it too much for an execution module to fire events to the salt-master bus? Are we too intrusive? The second solution seems lighter and less-coupled.

What do you guys think about this?
@terminalmage @rares-pop @skizunov @adelcast

[rares-pop]

The second option is a custom one that doesn't have to be upstreamed if there is no interest, but doesn't feel natural. From what I know, most if not all Pacakage Managers hold locks while installing and won't allow parallel installing/uninstalling. Remember the inotify beacon won't trigger when something changes, but is also based on a time interval that has to be set in the config (or changed dynamically at runtime).

I see the first option pretty generic and wouldn't be too messy to implement. Basically, have some config tokens for enabling this behavior (disabled by default), and some verbosity level (this verbosity can play the role of enabler too, and can mean - send an event for each package it installs, send an event for every percent of the operation that's completed, or every 5%, or 10% or whatever makes sense). I would try to implement this solution to see how it looks (how intrusive it is in the code), and how it affects scalability.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[rares-pop]

Any feedback/interest from saltstack on this?

[stale[bot]]

Thank you for updating this issue. It is no longer marked as stale.