On Monday, October 24, Apple released macOS Ventura, a major new update to the Macintosh operating system. Not long after, we began to receive reports from customers seeing that their real-time protection had turned off, and they were unable to turn it back on again. Malwarebytes software reported that it needed Full Disk Access – a special permission users can grant to programs that need it—yet reports said that Full Disk Access was already turned on.
Initially, we thought this was a flaw in our Malwarebytes 4.17 beta for Mac, which we had released the same day. Then reports started coming in from people using older versions of our software. We realized that the problem was far worse, and more widespread, than we’d initially thought. Our Monday had gone from bad to worse.
Some aspects of the problem were starting to seem like macOS bugs, though, so I reached out to friends at other companies. In no time, I had reports back from multiple people, all reporting exactly the same problems with their own products. As we collectively dug into the problem, it quickly became apparent that the bug had to be in macOS. Specifically, it seemed to be a bug in Transparency, Consent, and Control (TCC).
What is TCC?
Transparency, Consent, and Control, aka TCC, is a technology in macOS that prevents apps from accessing data that may be sensitive without explicit user consent. As an example, your Documents folder is one that could easily contain sensitive information, so on modern versions of macOS, apps must be given permission by the user in order to access files there. When an app attempts to access a file in the Documents folder, macOS will show an alert like this:
In the case of security software using Apple’s Endpoint Security framework, it is an Apple-enforced requirement that the software get a higher level of permission from TCC – namely, Full Disk Access. This is because security software generally needs to be able to access all files on disk, and Full Disk Access (FDA) is the permission that allows that.
To grant FDA, the user must open System Preferences (renamed to System Settings in Ventura), go to the Security & Privacy panel, and grant the software in question access within the Full Disk Access list. This is something third-party software cannot control, and can only guide the user through the process of granting this access.
Unfortunately, in Ventura, affected software will appear to have FDA within this settings pane, but in reality it does not. Worse, you cannot simply “turn it off and back on again,” as the switch for turning off FDA for the security software refuses to turn off. This leaves the software in an unfortunate state where it cannot function, and the user (seemingly) cannot give it the access it needs.
What caused the bug?
It all began with a bug in macOS that was presented by security researcher Csaba Fitzl at the Objective by the Sea conference in Spain (and that had been submitted to Apple by him many months earlier). The bug was almost ridiculously simple: Execute a simple, short command (tccutil reset All) in the Terminal and you could revoke Full Disk Access from all security clients installed on the machine, rendering their real-time protection features inactive.. (Sound familiar?)
According to Csaba, the fix for this bug had gone on for some time. Apple attempted to fix it repeatedly, and each time Csaba found a way to bypass the fix. Eventually, Apple decided to make more significant changes in an attempt to fix the bug, but at the time of the conference in early October, Apple had not released its fix, and Csaba had not been able to verify it.
As Twitter discussions of the Ventura bug unfolded, security researcher Mikey (@0xmachos on Twitter) announced a discovery that seemed to point the finger at Apple’s fix for Csaba’s vulnerability.
The TCC.db file is a database that maintains all the TCC permissions the user has granted to various apps. According to Mikey, it seems that Apple’s fix for the vulnerability involved assigning a new TCC entry for endpoint security clients, like Malwarebytes. Presumably, these would be exempt from the reset command involved in Csaba’s vulnerability.
However, what seems to have happened is that endpoint security clients that already had the older permission suddenly found themselves in possession of two permissions that do not play well together. For whatever reason, this left the endpoint security software in a state where it was not regonized by the system as having FDA, and System Settings was unable to allow the user to change that.
In essence, Apple’s “fix” for this vulnerability ended up causing the results of the vulnerability, for all security software on all Ventura systems. 🤦♂️
Adding to the confusion, it appears that endpoint security clients on Ventura are also granted additional permissions unexpectedly. These permissions are Input Monitoring (allows monitoring of keyboard input), Screen Recording (allows recording of the screen and audio), Accessibility (allows control of the computer), and Developer Tools (allows execution of software that would not normally be allowed).
Why these permissions are granted is anyone’s guess. Malwarebytes does not request or use any of these permissions, yet they’ve been granted to us without the user’s knowledge. There are some programs we detect as unwanted software that have gotten the rights from Apple to use the Endpoint Security framework. Imagine one of these programs pushing out an update to take advantage of these new permissions on Ventura, to do things like keylogging without the user being aware!
How can this be fixed?
Fortunately, Apple is aware of the issue and has said that it will be fixed in macOS Ventura version 13.1. Although Ventura 13.1 is in beta, it does not appear that the current beta contains the fix. I’ve tested the current 13.1 beta, and it does not solve the problem. Thus, at this time, I cannot verify the fix or describe how the fix will work.
Unfortunately, it’s anyone’s guess as to how soon Ventura 13.1 will be released, and many people are in need of a solution immediately. There is a workaround. We have posted an article with details for getting Malwarebytes working again, but the same solution works for any other endpoint security client. Essentially, this boils down to:
- Open Security Settings -> Security & Privacy -> Full Disk Access
- Select the security software in the Full Disk Access list
- Click the minus (-) button at the bottom of the list to remove it
- Try to turn on any real-time protection features in your security software
- The security software should reappear in the Full Disk Access list; flip the switch to give it Full Disk Access
Some details may vary slightly about exactly how all this works for the software you’re using, but in general, this should work.
This is the best and cleanest solution, but interestingly, was not the first solution I found. The first thing I tried was to actually execute Csaba’s vulnerability. In other words, I ran tccutil reset All in the Terminal on an affected machine. This fixed the problem, although it also had other side effects, so I don’t recommend it as a solution for others. In hindsight, it seems likely that this behaved exactly as Apple would have intended: It removed the old kTCCServiceSystemPolicyAllFiles permission from all apps—including Malwarebytes, in my case—and left the new kTCCServiceEndpointSecurityClient permission in place.
I’d also recommend that you revoke our ability to do Input Monitoring and Screen Recording. We don’t need and should not have those permissions. I’d also say that you should revoke our Accessibility and Developer Tools permissions, but in my testing, it appears that you can’t.
What’s the takeaway?
Developing software for Apple is like swimming with a whale. (I take no credit for this metaphor; I believe I heard it from a talk by Sal Soghoian.) I would imagine that swimming with a whale would be an amazing experience, and working with Apple can be the same. However, swimming with a whale is also a potentially dangerous experience. A whale is a huge, powerful animal, and if you find yourself in the wrong place at the wrong time, you could be injured or killed by a mere flip of fin or fluke.
Working with Apple is the same. Apple security researchers and developers love Apple’s Endpoint Security framework, and it has seen widespread adoption. However, it’s also the only game in town, and as this incident illustrates, a simple error on Apple’s part can send an entire industry reeling.
Although we at Malwarebytes are seeing a lot of praise from customers about our response to this issue, we’re also seeing a flood of refund requests, angry complaints, uninstalls, and the like. We’re seeing non-technical people struggle with the workaround. One week in and we have more than a thousand support tickets from affected customers, with only a small fraction of all customers on Ventura so far. As Ventura adoption rises, so will the problems – until, hopefully, Ventura 13.1 is released.
This issue was an accident. Apple was simply trying to do a good thing and fix a vulnerability. Nonetheless, it is going to have a huge impact on the entire security industry for months. It’s going to cost every company in the industry large amounts of money in customer support costs, refunds, and lost customers. It’s going to cause average people all manner of frustration, potentially even putting them in danger when their security software stops working.
Such are the dangers of swimming with the whale.
