10.11.6 Security Update (2016-001) breaks local admin logins

[jmahlman]

Alright, so this is a real noodle-scratcher here. We're completely lost as to what is going on so I figured I'd put it out there and maybe someone will be able to help.

Just upgraded all of our systems to 10.11.6 a few weeks ago using our process which we've been testing and using for months. We create our base images using AutoDMG and during first-run our admin account was created using this script:

#!/bin/sh
. /etc/rc.common
dscl . create /Users/macadmin
dscl . create /Users/macadmin RealName "macadmin"
dscl . passwd /Users/macadmin <PASSWORD>
dscl . create /Users/macadmin UniqueID 501
dscl . create /Users/macadmin PrimaryGroupID 80
dscl . create /Users/macadmin UserShell /bin/bash
dscl . create /Users/macadmin NFSHomeDirectory /Users/macadmin
cp -R /System/Library/User Template/English.lproj /Users/macadmin
chown -R macadmin:staff /Users/macadmin

We've apparently been using this method for some time now (I've only been here since December, and it's been in use much longer) and it's never been a problem.

This past weekend our systems ran Apple software updates and installed Security Update 2016-001 (we tested it locally and found it was fine, so we let it go out). We received word on Tuesday that another admin couldn't log into the macadmin account that was created. When attempting to log in the password would be accepted but the spinning wheel would not go away..it just sat there forever. The only way to stop it was to kill the loginwindow remotely or reboot the machine. This is occurring on EVERY machine where the update was pushed out.

Now for the confusing part! We found that running "rm -rf /Users/macadmin" from ARD allows you to log in magically. This is great, we don't use this account for anything except diagnostics and on machines without AD binding. We create a policy and scope it to a small lab that just runs that command. It runs fine, but we still cannot log in. After logging in with an AD account we see that the folder is removed and the account is still there. Pushing the command with ARD again allows us to log in. D:

So we decide to go farther and create a small script that completely removes the macadmin account:

#!/bin/sh
/usr/local/jamf/bin/jamf deleteAccount -username macadmin -deleteHomeDirectory

Add this to a policy that creates the account after this script runs (using the built-in account creation tool in the JSS) and it doesn't work. We tried using two policies (one to remove using the built-in JSS account tool and one to add using it also), we also tried using CreateUserPkg to create the account and it also didn't work. The accounts were created in both instances, but we were still unable to login. Going back to ARD and running the "rm -rf /Users/macadmin" command allows us to login again. Adding that line to the script doesn't help. !!!!

At this point we're beating our heads on the desk.

So finally we decided to create the macadmin account with the "jamf createAccount" command after the "jamf deleteAccont" command in ARD and it worked! Account gets recreated and I can log in, great! So I run both of those commands via Casper Remote to test again and, finally, that works too! Now we're on to something...

I created a policy with the following script:

#!/bin/sh
/usr/local/jamf/bin/jamf deleteAccount -username macadmin -deleteHomeDirectory
sleep 5
/usr/local/jamf/bin/jamf createAccount -realname macadmin -username macadmin -password <password>

And this worked! Somehow, that worked and using all other methods (which basically did the same steps) didn't.

I have so many questions here...

1. Why did the accounts die in the first place? We've never had an issue creating accounts in the past with the script (We've since moved to adding the account via the built-in method in Casper)
2. Why were certain commands only working when run in ARD and not in Casper Remote or in a policy?
3. Why didn't the built-in account creation and deletion work...but scripting them did?

Sorry for the novel, we were just so confused by this. I spent an entire day working on a solution to this...thankfully I fixed it this morning with the script above. Now to figure out WHAT cause this.

[dwandro92]

I would recommend commenting out the last two lines of the user creation script and see if that helps.

cp -R /System/Library/User Template/English.lproj /Users/macadmin
chown -R macadmin:staff /Users/macadmin

[jmahlman]

We're not using that script anymore (as of this week :D ). But I think I see why you recommend that.

[Nix4Life]

@jmahlman do you know how long after the update was deployed that this happened? we have been deploying it since last week with no issues, and I just imaged a computer and watched this update get puled down.

[jmahlman]

@LSinNY The update went on 9/2, we didn't get a call about it until 9/6. Mind you, updates didn't install until nighttime on Friday then we were off on Monday for Labor day.

It's not an account that we typically use; however, we have one tech that seems to like using it a lot and that's how it was found.

It doesn't seem to be affecting machines with local accounts created using System Prefs.

[flyboy]

Have you done a ls -ln on the /Users folder, and the contents of /Users/macadmin to make sure something weird is not going on with userids and permissions? The "-n" will force it to print the numeric userid and group id instead of the name. Do they match what's assigned to your account? It seems like a permissions thing since deleting the home folder allows you to login. It might be worth it to compare the home folder permissions when you can login, and when you can't. Also, If you've done any customizations to the default user template and not gotten the permissions right, something about the security update may not be as lazy about enforcing correct permissions as before. I'm thinking about like how you had to have permissions on the old StartupItems just right or they wouldn't load.

Also, are you bound to any kind of directory server? is there a chance that someone created a domain account called "macadmin" that may be causing a conflict?

[jmahlman]

@Berrier I'll have to take a look at the ls -ln output...we were thinking it could be a permissions issue also. We found that our office machines that have been updated do not show this problem..so we're trying to narrow it down to a DMG that uses FUT/FEU that's only found in our Lab configs.

Computers are bound to AD but there is no macadmin account...also macadmin can login again once it's recreated.

[londonbg]

We've had this issue too. It's a complete pain in the arse and waste of time to correct.

[jmahlman]

@londonbg I'm really glad to see that it's not an isolated issue with us. It was an entire day wasted trying to find a fix and we're still not sure what caused it.

[jmahlman]

Finally able to replicate the issue on my test machine even without the dscl account script (We were almost 100% certain it wasn't that). So, that's good at least. Now to dig through DMGs one by one!

[jmahlman]

More fun!

After realizing that I didn't set the macadmin account as an admin in my script, I re-ran it with the admin flag and lo and behold, it cannot login again unless you remove the home directory after trying to log in once.

We've also confirmed that it's happening with ALL admins on the system, even those set in a network group. So if your group was designated as an admin on that machine, you cannot log-in.

@londonbg Did you figure anything out? What caused it? How did you fix it? How widespread is it? We're really ripping our hair out over this.

[jmahlman]

Checking the logs I noticed something repeating over and over again when logging in:

unknown com.apple.xpc.launchd[1] (com.apple.mbbackgrounduseragent): Service only ran for 0 seconds. Pushing respawn out by 10 seconds.

Looking online I came across this page and tried what he suggested and what do you know, I can login again.

I don't know what mbbackgrounduseragent is, but unloading the service SEEMS to work. Anyone have insight here?