Minions don't give good error messages when Master is unable to save their keys

[goetzk]

Description of Issue/Question

When running with a read only /etc/ on the salt master it is unable to write out records of the clients connection (mentioned in #37798). As a result the client is not authenticated. I don't know if its possible, but it would be good to have a more meaningful error on the minion than the backtrace displayed at the moment.

Setup

I have a freshly installed salt setup with one master and one minion, using the manual set up in the getting started docs. The packages are the latest installed install_salt.sh

 dpkg -l |grep salt
ii  salt-common                          2016.3.4+ds-1                 all          shared libraries that salt requires for all packages
ii  salt-master                          2016.3.4+ds-1                 all          remote manager to administer servers via salt
ii  salt-minion                          2016.3.4+ds-1                 all          client package for salt, the distributed remote execution system

docs: https://docs.saltstack.com/en/getstarted/fundamentals/install.html

Steps to Reproduce Issue

(Include debug logs if possible and relevant.)

• setup per Salt packages fail on masters with read only /etc/ #37798 with read only /etc
• attempt to add new minion
  An error like this will occur in the minions logs:

2016-11-21 21:26:08,997 [salt.scripts     ][WARNING ][17820] Fatal functionality error caught by minion handler:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/scripts.py", line 128, in minion_process
    minion.start()
  File "/usr/lib/python2.7/dist-packages/salt/cli/daemons.py", line 359, in start
    self.minion.tune_in()
  File "/usr/lib/python2.7/dist-packages/salt/minion.py", line 2017, in tune_in
    self.sync_connect_master()
  File "/usr/lib/python2.7/dist-packages/salt/minion.py", line 938, in sync_connect_master
    raise six.reraise(*future_exception)
  File "/usr/lib/python2.7/dist-packages/tornado/gen.py", line 876, in run
    yielded = self.gen.throw(*exc_info)
  File "/usr/lib/python2.7/dist-packages/salt/minion.py", line 947, in connect_master
    master, self.pub_channel = yield self.eval_master(self.opts, self.timeout, self.safe)
  File "/usr/lib/python2.7/dist-packages/tornado/gen.py", line 870, in run
    value = future.result()
  File "/usr/lib/python2.7/dist-packages/tornado/concurrent.py", line 215, in result
    raise_exc_info(self._exc_info)
  File "/usr/lib/python2.7/dist-packages/tornado/gen.py", line 876, in run
    yielded = self.gen.throw(*exc_info)
  File "/usr/lib/python2.7/dist-packages/salt/minion.py", line 593, in eval_master
    raise exc
SaltClientError: Attempt to authenticate with the salt master failed with timeout error
2016-11-21 21:26:08,999 [salt.scripts     ][WARNING ][17820] ** Restarting minion **

[Ch3LL]

ping @cachedout or @DmitryKuzmenko just wondering if either of you think it is even possible to query this kind of data? Just not sure if we could log information like @goetzk 's requesting when there is a timeout error on the minion side. Will approve as a feature request if you believe its possible. thanks

[DmitryKuzmenko]

Agree this should be handled. Moreover there is the following message in master log:

Future exception was never retrieved: Traceback (most recent call last):

That means this case isn't handled properly in our code.

I think we should wrap all IO operations in AESReqServerMixin._auth with try/except and return a specific ret value for such case that would be handled in salt.crypt.AsyncAuth.sign_in as a 'Server error' case with a message like 'Check master log'.

[Ch3LL]

Perfect thanks for the feedback @DmitryKuzmenko ! Will approve this as a feature request then :)

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[goetzk]

Seems silly to have a bot close feature requests... unless the feature was completed.

[rallytime]

@goetzk We are always happy to open these again. :)

[stale[bot]]

Thank you for updating this issue. It is no longer marked as stale.

[stale[bot]]

Thank you for updating this issue. It is no longer marked as stale.

[goetzk]

Thanks ^_^

…

On 6 August 2018 12:28:40 am AEST, Nicole Thomas ***@***.***> wrote: @goetzk We are always happy to open these again. :) -- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: #37799 (comment)

-- Sent from my Android device with K-9 Mail. Please excuse my brevity.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[stale[bot]]

Thank you for updating this issue. It is no longer marked as stale.