Skip to content

Conversation

r3lgar
Copy link

@r3lgar r3lgar commented Mar 25, 2017

20 is too much tries (sensitive for CPU and network);
For 5 seconds we don't have enough time to fix the problem (don't flood the log);

20 is too much tries (sensitive for CPU and network);
For 5 seconds we don't have enough time to fix the problem (don't flood the log);
@derlaft
Copy link
Owner

derlaft commented Mar 25, 2017

20 is too much tries (sensitive for CPU and network);

Usually, managers set up a progressive scale. Should we do the same?

The more restarts we have, the more we wait.

@r3lgar
Copy link
Author

r3lgar commented Mar 25, 2017

Usually, managers set up a progressive scale. Should we do the same?

Yes, but I don't understand how manage it per process/script.

The more restarts we have, the more we wait.

If uptime of the process/script is stable (10 mins? 1 hour?), we must decrease the timer.

And I think, if process/script exits with 0 (no error), we don't need to increase the timer.

@derlaft
Copy link
Owner

derlaft commented Mar 25, 2017

Yes, but I don't understand how manage it per process/script.

I would make it smth like this for decrementing:

go func(savedRestarts int) {
  var interval = func(int) time.Duration {.... }
  time.Sleep(interval(savedRestarts))
  if savedRestarts == restarts {
    restarts --
  }
}(restarts)

@r3lgar
Copy link
Author

r3lgar commented Mar 25, 2017

restarts

You can create an array at startup and append restart for each process/script and manipulate them. Is this possible in Go?

@derlaft
Copy link
Owner

derlaft commented Mar 25, 2017

@r3lgar ofc, but there's no need, you can just use any local variable as a storage in the worker func.

@r3lgar
Copy link
Author

r3lgar commented Mar 25, 2017

@derlaft but why? It's more flexible by using restarts/waits per process/script because of one process/script can hangs all of them. This logic prevents such fuckups.

@derlaft
Copy link
Owner

derlaft commented Mar 25, 2017

@r3lgar

Look at https://github.com/derlaft/r3lgar-session/blob/master/main.go#L32

All the variables declared in this block are saved between restarts

@r3lgar
Copy link
Author

r3lgar commented Mar 25, 2017

All the variables declared in this block are saved between restarts

We can add timeout (or something like that) to them and increase/decrease it. On start, if process started successfully, we can change timeout to default (0?), otherwise increment it and wait. For problemaic scripts it's a good workaround, am I wrong?

As example: we has broken script, we stoped the daemon, fix script, start daemon again, if it successfully starts the script, all is OK, continue the job as usual, if not, wait; less flood to log/stderr. Not simple but doesn't requires the configuration file for such workarounds.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants