What’s New in macOS Big Sur Security

Underneath the distinctive new look of macOS Big Sur are changes in security architecture which build on those in Catalina: existing division of the startup volume into two is enhanced by even greater protection for the system; notarization is enforced more rigorously without blocking the use of unsigned code; and macOS moves away from extensions running in kernel space towards user space extensions, including special Endpoint Security Extensions.

Sealing the system

The biggest single change in macOS 11 is its new Sealed System Volume (SSV), which replaces the separate System volume introduced in macOS 10.15. This deepens system protection from the existing read-only volume covered by System Integrity Protection (SIP).

During macOS installation, once its System volume has been installed, cryptographic hashes are computed for every component on that volume and assembled into a tree (like a Merkle tree), culminating in a single, master hash termed the Seal. Those hashes are saved as metadata and a file system snapshot is made of the volume. Instead of macOS mounting the System volume read-only as it does in Catalina, only that sealed snapshot is mounted, giving immutable system files further robust layers of protection from tampering and error. This mechanism also protects against failed system updates, whose Seal won't match the prescribed.

During early startup, macOS Big Sur checks the Seal on the system. If that's broken, the operating system won't boot and has to be reinstalled. Recovery mode offers an option to disable that check, making it possible to customize a System volume and run it unsealed; setting that up is intricate and non-trivial.

Once unsealed, users can't reseal the system, and the only ways of creating a sealed system are using a macOS Big Sur installer or updater, or with the Apple Software Restore command tool asr. Previous methods of copying or cloning the System volume no longer produce a bootable result, and compatible third-party utilities must also use asr to be successful.

macOS Big Sur provides a Sealed System Volume that raises the protection of key system files beyond the reach of all current malware and should withstand the most determined attacker from altering them after the OS has booted. It also guards against inadvertent corruption and guarantees system integrity.

Extensions

Mutable system files are still stored on the writable Data volume, and not protected by sealing, nor other measures applied to the majority of the system. Among those mutable files are any user-installed kernel extensions, which some had been expecting would be blocked in macOS Big Sur. While macOS Big Sur is more pernickety overloading some older extensions, Apple has delayed a complete ban to allot developers more time to migrate from their reliance on extensions running in the kernel space and replace with System Extensions in the user space.

Extensions are needed by apps that alter or extend features implemented in the kernel and the over 300 standard kernel extensions provided in macOS. Classic purposes include device drivers to support peripherals, network monitoring including software firewalls, DNS proxies and VPN clients, tracking changes made in the file system and support for additional file systems.

When Big Sur's kernel and kernel extensions have loaded during startup, memory pages in kernel space are locked by Kernel Integrity Protection (already used in iOS) to prevent their modification. As System Extensions run in user space, their access to the kernel and its features is strictly controlled. One System Extension class of particular value in security is the Endpoint Security Extension, which can monitor and authorize events such as process execution and forking, file system events including file manipulation, access to file system metadata and the connection of sockets. As with all System Extensions, these require a special entitlement granted exclusively by Apple, and their installation and control is managed by their companion app.

The Endpoint Security framework is already proving valuable for implementing proactive security tools that aren't dependent on looking for known malware, but can detect potentially malicious behavior and watch vulnerable parts of the system which still have to be installed on the Data volume.

Moving away from extensions running in the kernel space brings a substantial reduction in attack surface, as well as improving system reliability by eliminating the risk of conflicts arising with third-party kernel extensions.

App security

Although there's no overall change in security requirements for apps and other third-party software, notarization is more strictly enforced, with users having to negotiate a sequence of two dialogs before newly-installed apps that aren't notarized can be opened. In Catalina, some users have learned that opening a new app in the Finder runs that app from a single dialog even when it isn't notarized. Within macOS Big Sur, this action is made more deliberate, as users must use the Open command a second time before being asked if they really want to run the app despite its lack of notarization.

Unsigned code can still be run on Intel models, but Apple Silicon Macs require all executable code (except scripts) to be signed. Although, that can just be with a locally generated ad hoc signature.

Apple’s new Sealed System Volume is a big step forward in securing the macOS system and has significant consequences for some users. Coupled with improved protection of kernel space by moving user extensions into user space, it makes macOS 11 significantly more resilient.