Need guard against/recovery from sssd failing to restart

[mberkowski]

Too many sequential calls to realm permit can cause a race condition where realmd tries to restart sssd multiple times. Systemd will detect it as failure and can't start it up.

That means NO ONE can log into the machine, it is dead to ssh on the network.

We need some way to recover from this. Suggestions:

• Plant something in a systemd unit override to ensure that sssd is always running, via Restart=always or similar
• Hosting support should probably implement this themselves
• A root cron that just looks for sssd running every minute and starts it if not

What we cannot do:
Restart sssd with ansible. You can't login to begin with.

Edit: see also (courtesy of colin) SSSD/sssd#6219

[cmcfadden]

It sure seems like this should just be a systemd managed thing - have you seen any info about why that'd be a bad idea?

[mberkowski]

No I haven't looked yet. I remain pretty convinced that we should only interact with sssd via realm permit and not dare modify the config file then restart (which would be a more normal way for ansible to do this)

[pandackindley]

I recommend experimenting with modifying the config file and restarting, if that's essentially what realm does. Bonus, since realm doesn't give super helpful exit codes on failure, Ansible might be happier modding the config file!

[mberkowski]

I have to do some research on how /etc/sssd/conf.d/* get applied. Because we can't/shouldn't assume simple_allow_users is not already populated with something and because simple_allow_groups is definitely already populated when we start on the machine, we need to be able add to it without breaking it. It's for this reason I favor calling realm permit, but that is clearly not meant to be done a bunch of times in a row.

[mberkowski]

Done some testing in modifying snippet files with /etc/sssd/conf.d/* - Docs are pretty limited on this - all we get is:

CONFIGURATION SNIPPETS FROM INCLUDE DIRECTORY
The configuration file sssd.conf will include configuration snippets using the include directory conf.d. This feature is available if SSSD was compiled with libini version 1.3.0 or later.

Any file placed in conf.d that ends in “.conf” and does not begin with a dot (“.”) will be used together with sssd.conf to configure SSSD.

The configuration snippets from conf.d have higher priority than sssd.conf and will override sssd.conf when conflicts occur. If several snippets are present in conf.d, then they are included in alphabetical order (based on locale). Files included later have higher priority. Numerical prefixes (01_snippet.conf, 02_snippet.conf etc.) can help visualize the priority (higher number means higher priority).

The snippet files require the same owner and permissions as sssd.conf. Which are by default root:root and 0600.

In testing this, if I drop a snippet like the following in there:

[domain/ad.umn.edu]
simple_allow_groups = another-group

This does not add to the existing groups that ship with the VM in its default state. It overwrites groups who are required to have basic access (the institutional admins, departmental admins). Those must be all included in the same line but that means we would need some way for ansible to capture that initial state so it could refer to it when modifying sssd.conf or adding snippets later. I do not like this idea very much because it is dangerously error prone. Some implementations:

• use lineinfile to match that line on first pass and comment it out along with some other identifier like # UMNANSIBLE simple_allow_groups =.... And then on subsequent runs the contents of that part go with whatever groups/users we add
• Same idea, but grep those values out to a file that stays on the box but doesn't get loaded by sssd and combine with additional users/groups when ansible runs.

Both of these give me hives. Who's got another idea?

[cmcfadden]

Agree that those options don't feel great, and I'd lean towards using realm with a systemd change in lieu of those.

Alternative approach would be to bake in the default groups (OIT-LPT) to the role with a flag about whether or not to include them (defaulted to true). Then we could safely plop in a .conf file with all of the right values and not worry about the initial sssd.conf state. (If I understand correctly)

It'd then be incumbent on the platform team to work with the community to revise the role if something changes in the future, but that's not necessarily a bad thing.

[mberkowski]

The "default" groups are the lpt group but also the departmental cesi admins group, so default state is variable by unit but important not to mess up lest the department lose access to the vm.

[cmcfadden]

The "default" groups are the lpt group but also the departmental cesi admins group, so default state is variable by unit but important not to mess up lest the department lose access to the vm.

But if folks are running ansible under tower, even if they hosed their dept groups, tower would be able to reconnect and fix that right? I think it'd be reasonable to expect folks to manage those groups in their roles.

[mberkowski]

As a stopgap, I went ahead and committed 8649464 to modify sssd.service. I am still nervous about mucking around in sssd.conf. For the time being, this role now provides a backstop against sssd crashing out from realm permit but I am going to leave this issue open for possible future revision of the whole procedure.