salt-ssh missing custom grains

[Cptn727]

Due to a firewall snafu, some of my minions lost connectivity to the salt-master today, but I could still ssh from the master to the minion. So I tried to use salt-ssh but discovered that custom grains were missing.

To illustrate the problem, I have reproduced it on a minion which is not having the connectivity issue, so I can show the comparison to the output from the regular salt command. Summarily, the issue is that default grains (id, fqdn_ip4, etc.) are present, but custom grains set in /etc/salt/grains are not.

For reference, these are the custom grains:

root@CT1-ETSYSADMINPROXY1:~#  salt-ssh -i CT1-ETCASSDEV1 cmd.run 'cat /etc/salt/grains'
CT1-ETCASSDEV1:
    dc: cn1
    dom: ef_com
    env: dev
    et_role:
    - cass

Here is what happens with salt-ssh:

root@CT1-ETSYSADMINPROXY1:~#  salt-ssh -i CT1-ETCASSDEV1 grains.get dom
[CRITICAL] Pillar render error: Rendering Primary Top file failed, render error:
Jinja variable 'dict object' has no attribute 'dc'
CT1-ETCASSDEV1:
root@CT1-ETSYSADMINPROXY1:~#  salt-ssh -i CT1-ETCASSDEV1 grains.get id
[CRITICAL] Pillar render error: Rendering Primary Top file failed, render error:
Jinja variable 'dict object' has no attribute 'dc'
CT1-ETCASSDEV1:
    CT1-ETCASSDEV1

and here is what happens with salt

root@CT1-ETSYSADMINPROXY1:~#  salt CT1-ETCASSDEV1 grains.get dom
CT1-ETCASSDEV1:
    ef_com
root@CT1-ETSYSADMINPROXY1:~#  salt CT1-ETCASSDEV1 grains.get id
CT1-ETCASSDEV1:
    CT1-ETCASSDEV1

Here is the versions report (identical for minion and master):

                  Salt: 2015.5.2
                Python: 2.7.6 (default, Jun 22 2015, 17:58:13)
                Jinja2: 2.7.2
              M2Crypto: 0.21.1
        msgpack-python: 0.3.0
          msgpack-pure: Not Installed
              pycrypto: 2.6.1
               libnacl: Not Installed
                PyYAML: 3.10
                 ioflo: Not Installed
                 PyZMQ: 14.0.1
                  RAET: Not Installed
                   ZMQ: 4.0.4
                  Mako: 0.9.1
 Debian source package: 2015.5.2+ds-1trusty1

Not sure if this matters at all, but the master is also a syndic.

[jfindlay]

@Cptn727, thanks for the report. An ssh minion is designed to operate out of a single directory under /tmp, so I would be surprised if it went looking in /etc/salt for custom grains. I think you will need to do a salt-ssh -i CT1-ETCASSDEV1 saltutil.sync_grains to send the custom grains down to that minion in the ssh role.

[basepi]

saltutil.sync_grains will not actually work in this instance. I think it would probably sync down custom grains from /srv/salt/_grains, but /etc/salt/grains is different, as it's not necessarily provided by the master. I'm thinking the best solution for interacting with minions that are part of a master cluster but have lost connection, is to have an option to look for those files at execution time.

[Cptn727]

@jfindlay thanks, I found that the saltutil.sync_grains did not make any change, the command outputs remained identical to those provided above. Hope that's useful feedback.

@basepi thanks. Our /etc/salt/grains file is generated at the same time as the salt-minion is being installed on a new box (via salt-call grains.setvals) but maybe there's some way we can adjust this to work for /srv/salt/_grains instead.

Also, I may have not been fully clear about it, but this issue is independent of the connectivity. We noticed it because of the connectivity issue, but it exists on all of our local-custom-grains-enabled minions with salt-ssh.

Our firewall issue has been corrected, so this is no longer impacting us, but it would be nice to have in case of a future issue. Let me know if I can provide any additional information you might want, or if there's a more "proper" way to rely on salt-ssh for fallback.

Thanks!

[basepi]

Nope, you're using salt-ssh fine. The fallback use case with static grains in /etc/salt/grains is just one we haven't thought about, and we need to address. Will probably just be a config value we support via a command line flag (and master config) to have the minions look for grains at that location.

[Semyazz]

I'd also like to see this feature working. I had to do a sort of workaround and cat /etc/salt/grains and then inject it back to $thin_dir/grains to make it available to salt-ssh

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[crhan]

This problem still exists!

[stale[bot]]

Thank you for updating this issue. It is no longer marked as stale.

[basepi]

@saltstack/team-triage

[nergdron]

I'm not sure if this is the same issue, but I ran into a problem today where custom grains aren't being run on minion when using salt-ssh. we've got a couple little local ones in our codebase's _grains folder, but they don't get evaluated, even when running salt-ssh 'hostname' saltutil.sync_grains first.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[p6knewman]

This would still be nice to have 🙏

[stale[bot]]

Thank you for updating this issue. It is no longer marked as stale.