MacBooks randomly becoming unmanaged

[spowell01]

Hey everyone, we have had an odd ongoing issue for a little while now and it seems to be getting worse lately. Random MacBooks are becoming unmanaged and we have to run the quick add package in order to bring them back under management. These are machines that have been managed for any period of time, some months and others just weeks. Just had about 4 students bring their laptops in because they were unable to install firefox from self service. It was giving the message that the machine was not managed and they should run a local recon. Obviously recon does not work since it won't recognize any jamf commands. We are now on JSS 9.21 but have seen the issue in previous versions as well. Anyone else dealing with something like this?

[scottb]

Are they admin's on the Macs? If so, it would take 2 minutes on google to figure out how to "sudo jamf removeFramework".
I'm new to Casper, but I imagine there's been a bunch of solutions to things like this. Our problem is we allow all users to be admins and some of them are smart enough to turn this stuff off or just delete it.

Are there any remaining bits of the jamf software/binary on those Macs? Because broken and removed are different issues.

[spowell01]

well i would say 80% of the time this occurs on student machines( mainly because we have way more of them). Our students are not administrators and would not be able to unmanage via terminal. Our teachers are full admins, but I'm very confident that the teachers are not manually removing the jamf framework, and thus requiring them to walk down to our office for a fix. I'll have to dig a bit deeper on the next example mac i have as far as the remaining framework....I say unmanaged because thats what the JSS states for the computer records. Self service remains in the dock, and i would imagine when removing the framework, self service goes along with it.

[mm2270]

I'm not sure that Self Service gets deleted when you run the removeFramework command. Perhaps but I never took notice if that's the case.
Let me ask - are these mostly or all MacBook Airs or new MacBook pros that don't have built in Ethernet? if so, are people/students sharing dongles perhaps? Just a thought.

[spowell01]

Nope, these are all 2011 MacBooks and macbook pro's. We have not provided any ethernet dongles on the few airs and new macbook pro's we have deployed, so thats not an issue. I will have to test and see if SS stays or goes when the remove framework command is given

Regardless Im sure that these kids are not removing the jamf framework....just had 8 more students come in between my last post and now. All the same issue.....im starting to think that the upgrade to 9.21 has caused this to be more of an issue.

[spowell01]

So I've now got an example machine in front of me and will be contacting my jamf rep soon here to maybe get to the bottom of whats happening. This particular machines record in the JSS still says managed, but it has not " checked in" in about a month. Now it has successfully ran policies as recent as yesterday, yet anything in self service fails and jamf commands are not recognized via terminal. Heres a screenshot of the JSS record showing that its still managed

external image link

[spowell01]

hmm, not sure why the image wouldn't link, but i just checked for the jamf binary in /USR/SBIN and all i was able to find is the jamfagent file. Where else would i look for specific jamf files?

[mm2270]

The jamf binary and the jamfAgent both live in /usr/sbin/ If the regular jamf binary isn't there, then that's at the root of the problem and you'll need to figure out what's removing that. Can't be user's if they aren't local admins because /usr/sbin/ is protected from non admins.
Any new Antivirus software recently that could be blasting that? Can you do a search on the Mac to see if the binary was perhaps moved to another location?

[spowell01]

Well the Jamf support guy had me run a quick add package on the example machine i had so its no longer a valid test.....No antivirus software at all here....Whenever i get another example machine in hand I'll try a terminal search for the jamf binary to see if its been moved.

Also, he has me running the recon tool systematically on our entire network. I guess this is essentially re-enrolling every mac that it can touch, which isn't much of a solution IMO.

[spowell01]

I've examined 3 other machines just this afternoon that all were showing managed in the JSS, but hadn't checked in, in about a month. One was showing binary version 9.11 and two others were showing binary version 9.12. The last policy I see is that they all updated inventory, and then apparently lost all communication with the JSS. The jamf binary is actually totally gone from the machine, not just moved.

[bentoms]

@spowell01, So maybe when the clients are contacting the JSS & trying to update the JAMF Binary etc...

Something's interfered with the process to upstate the binary, resulting it bits missing??

Maybe it's been an issue with older v9 versions?

[damienbarrett]

I've seen this kind of thing happen when I assign a "Loaner" machine to a student because his/her machine needs to be repaired. Since moving to MacBook Airs in Sept. and 10.8.x, it happens almost every single time. I'm sure it's related to the fact that I'm just swapping SSDs between the two machines, but that act breaks the management of the machine. It also causes the machine name to be blank.

Until l can figure out why, I've simply worked into my repair routine steps to run QuickAdd and force management.

[mm2270]

@bentoms][/url may be correct. Perhaps at a point when the binary is being updated after checking in to the JSS, somehow the process is being interrupted. I would think that the process would work in a way that the old binary remains intact until the new one is copied down, confirmed, then the swap happens, but I actually don't know much about how that upgrade process works.

Other than that dude, it sounds like you've got a serious issue on your hands. I have never seen a case of the jamf binary being removed from a system other than when running the remove framework command, or someone manually deleting it from /usr/sbin/ Since we know its very unlikely to be users doing this, I don't know where that leaves you other than a possible bug that JAMF may need to address. But I wouldn't rule out some environment variable here as well. Something in your environment may be causing this. I just don't know what. :/

Hope you figure it out.

[bentoms]

@damienbarrett v9 right? If so, then the devices UUID is used to distinguish between devices.

When changing SSD's there with be a mish mash locally & on the JSS which is probably why you're seeing an issue.

[blackholemac]

Hate to chime on this as I'm guessing spowell01 may already know this, but when dealing with MacBook Airs, MacBooks or any other device that uses USB or Thunderbolt to Ethernet dongles, don't forget to exclude the removable dongle's MAC address in Casper. If you don't it reaks havoc on inventory records and entries which may cause a machine to not be managed properly...it's an idea...may not be the one that solves this.

[spowell01]

Thanks for the responses everyone.

@blackholemac Thanks but we already know about the ethernet dongles and stash them away right after imaging, as well as add their mac to the exclusion list.

Here is an excerpt from the jamf.log on an affected client. I have sent it over to jamf, but its quite obvious that its failing to update various pieces of the jamf infrastructure and then just falling on its face.
Mon Oct 07 09:52:05 MSDMB07 jamf[81]: Network state changed, checking for policies...
Network state changed, checking for policies...
Mon Oct 07 09:52:05 MSDMB07 jamf[3209]: Checking for policies triggered by "networkStateChange"...
Mon Oct 07 09:52:05 MSDMB07 jamf[3217]: Checking for policies triggered by "NetworkChange"...
Mon Oct 07 09:52:07 MSDMB07 jamf[3209]: Upgrading jamf binary...
Mon Oct 07 09:52:07 MSDMB07 jamf[3217]: Upgrading jamf binary...
Mon Oct 07 09:52:09 MSDMB07 jamf[3209]: The management framework will be enforced as soon as all policies are done executing.
Mon Oct 07 09:52:09 MSDMB07 jamf[3217]: Unable to move /private/tmp/jamf to /usr/sbin/jamf: Error Domain=NSCocoaErrorDomain Code=4 "“jamf” couldn’t be moved to “sbin” because either the former doesn't exist, or the folder containing the latter doesn't exist." UserInfo=0x4866b0 {NSUserStringVariant=( Move
), NSFilePath=/private/tmp/jamf, NSDestinationFilePath=/usr/sbin/jamf, NSUnderlyingError=0x4865e0 "The operation couldn’t be completed. No such file or directory"}
Mon Oct 07 09:52:09 MSDMB07 jamf[3209]: Upgrading jamf agent...
Mon Oct 07 09:52:09 MSDMB07 jamf[3217]: Upgrading jamf agent...
Mon Oct 07 09:52:10 MSDMB07 jamf[3217]: Upgrading jamfHelper.app...
Mon Oct 07 09:52:11 MSDMB07 jamf[3209]: Unable to move /private/tmp/jamfAgent to /usr/sbin/jamfAgent: Error Domain=NSCocoaErrorDomain Code=4 "“jamfAgent” couldn’t be moved to “sbin” because either the former doesn't exist, or the folder containing the latter doesn't exist." UserInfo=0x46d550 {NSUserStringVariant=( Move
), NSFilePath=/private/tmp/jamfAgent, NSDestinationFilePath=/usr/sbin/jamfAgent, NSUnderlyingError=0x46d2f0 "The operation couldn’t be completed. No such file or directory"}
Mon Oct 07 09:52:11 MSDMB07 jamf[3209]: Upgrading jamfHelper.app...
Mon Oct 07 09:52:11 MSDMB07 jamf[3209]: Unable to move /private/tmp/jamfHelper.app to /Library/Application Support/JAMF/bin/jamfHelper.app: Error Domain=NSCocoaErrorDomain Code=516 "“jamfHelper.app” couldn’t be moved to “bin” because an item with the same name already exists." UserInfo=0x18b1660 {NSUserStringVariant=( Move
), NSFilePath=/private/tmp/jamfHelper.app, NSDestinationFilePath=/Library/Application Support/JAMF/bin/jamfHelper.app, NSUnderlyingError=0x18b15b0 "The operation couldn’t be completed. File exists"}
Mon Oct 07 09:52:11 MSDMB07 jamf[3217]: Upgrading Self Service.app...
Mon Oct 07 09:52:11 MSDMB07 jamf[3209]: Upgrading Self Service.app...
Mon Oct 07 09:52:13 MSDMB07 jamf[3217]: /private/tmp/Self Service.app/Contents/MacOS/Self Service cannot run on this machine. It will not be upgraded.
Mon Oct 07 09:52:13 MSDMB07 jamf[3209]: The checksum of /private/tmp/Self Service.app/Contents/MacOS/Self Service did not match. (57c4f87a553648134d34a94f125348d6 != )
Mon Oct 07 09:52:13 MSDMB07 jamf[3217]: Executing Policy Fix Software Update Server ON NETWORK...
Mon Oct 07 09:52:13 MSDMB07 jamf[3209]: Adding launchd task com.jamfsoftware.task.checkForTasks...
Mon Oct 07 09:53:22 MSDMB07 jamf[81]: Network state changed, checking for policies...
Network state changed, checking for policies...
Mon Oct 07 09:53:22 MSDMB07 jamf[81]: Error executing command: ( "/usr/sbin/jamf", policy, "-event", networkStateChange
)
Mon Oct 07 10:13:13 MSDMB07 jamf[81]: Network state changed, checking for policies...
Network state changed, checking for policies...
Mon Oct 07 10:13:13 MSDMB07 jamf[81]: Error executing command: ( "/usr/sbin/jamf", policy, "-event", networkStateChange
)
Mon Oct 07 10:56:52 MSDMB07 jamf[81]: Network state changed, checking for policies...
Network state changed, checking for policies...
Mon Oct 07 10:56:52 MSDMB07 jamf[81]: Error executing command: ( "/usr/sbin/jamf", policy, "-event", networkStateChange
)

[bentoms]

@spowell01. Woohoo, looks like I was right.

Sadly I have no idea how to help further. :(

I'm sure JAMF Support will figure it out, please post here when sorted.

BTW, what OS is the JSS on? Do you have tight firewall policies?

[spowell01]

We are running our JSS on a virtual server with 2008 R2 as the OS. We actually don't have any firewalls turned on in our environment, but do have an ISP provided and managed Palo Alto firewall that sits outside our enviroment.

I have gathered logs and have a scheduled call on monday with our Jamf rep and one of his colleagues. Hopefully we can get to the bottom of whats going on here.

[dpecht1221]

@spowell01 did you ever find out what was causing your MacBooks to become unmanaged and reach a resolution? I am having the same thing happen to hundreds of computers.

[spowell01]

sorry depcht, I dont recall the exact solution on this one, but we worked pretty heavily with JAMF support. I believe the root of the issue was stemming from a bugged Jamf binary upgrade. Essentially the machines were supposed to delete their old binary and then lay down the new one, we think it wasnt deleting the old binary which casued issues when laying down the new one. The end result was that the machine would not be managed. We are currently on 9.61 and we haven't seen this issue with the last couple upgrades.

[alexjdale]

We've had a machine go unmanaged from time to time, looks like it is happening during a bad binary upgrade (happens after we upgrade the server). It's rare, we have 10k systems and I see one every week or two. Looks like we lose all data for that system. Our auto-repair scripts would catch that and re-enroll them, but we would lose any escrowed FV key.

[scottb]

I freaked out after following this as I had one yesterday go unmanaged - turned out to be a bad imaging job. After a rebuild, it was fine. Is there any way to scope for unmanaged Macs? I was looking around (9.32) and could not find a way to locate them. I'm sure I'll be embarrassed at the answer...

[mm2270]

@boettchs - You can get notified when a Mac or Macs fall into or out of the All Managed Macs Smart Group. But you'll likely get spammed with emails with this setting. Its really the only effective way though. By their very nature, only a Managed Mac can be in a Smart Group, so you can't create a Smart Group of Macs that are Not Enrolled or Not Managed. Only Advanced Searches can locate those.

[scottb]

@mm2270 - thanks, glad that was the case. Email spam is also happening with Restricted software - three per each violation. In the end I can live with it I guess.

[msnowdon]

@blackholemac I know this is an old thread I stumbled upon but how/why do you need to exclude USB/Thunderbolt MACs in Casper?

Thanks

Mark

[bentoms]

@msnowdon Pre-JSS v9, the JSS used to use MAC addresses to track computers.

This caused an issue with removable MAC's from things like Ethernet adaptors.

With v9, JSS moved to UUID of the Mac so this is not needed.

[blackholemac]

The process isn't terribly relevant anymore, but that being said, I'll explain my logic at the time. You can still exclude MAC addresses but the problem isn't quite as acute.

It used to be that JAMF used the MAC address of the device as it's index record. So imagine if you were imaging 400 MacBook Airs and used 40 thunderbolt to Ethernet dongles to do it. Used to be that the database would report the MAC address of your imaging dongle as the MAC address of the device and each device you add would overwrite another one. Less than helpful. You would exclude the MAC address of your imaging dongle so it wouldn't report it's info to Casper. Page 440 of the Casper Admin Guide covers this particular issue.

That being said, newer versions of hardware/Casper implementations no longer rely on MAC address as an identifier...instead the UDID is used. I think this change came in Casper 9.0, BUT any existing records from before may include it.

So, these days I'm not to worried about it, but I usually take a second to exclude MAC addresses from my imaging dongles at the very least.

[msnowdon]

Oh, okay thanks. Came across this article because MacBooks are suddenly becoming Unmanaged and I was trying to figure out why. I am now looking into RTrouton's auto repair process:

https://derflounder.wordpress.com/2014/04/23/caspercheck-an-auto-repair-process-for-casper-agents/

Thanks

Mark