Spotlight and iCloud Drive headbang in the CPU mosh pit

I have a Mac Studio M2 Max with 32 GB of RAM and a 1 TB SSD in front of me. While not the most powerful personal computer you can purchase today, it is among the most computationally capable ever made—a vastly capable machine. So why do I have to wait for typing to catch up with me and see a rainbow spinner on the regular?

I’ll tell you why: iCloud Drive and Spotlight. Apple lacks any transparency about how iCloud Drive works or fails. The same is true of Spotlight. These big black boxes churn away, performing a million billion operations a day, and we don’t know exactly what they do. That’s not me being conspiratorial: I honestly don’t care what’s happening under the hood—until it breaks. Without diagnostic tools, even someone like me, ostensibly an expert, can be completely at a loss and waste many hours.

I wrote “Cloudy with a Chance of Insanity: Unsticking iCloud Drive” for TidBITS back in October 2023 about a multi-month odyssey with Apple technical support in trying to dislodge unsyncable files.1 That problem has barely surfaced since—an 80K file will sometimes get stuck for a while—and I thought Apple might have really solved syncing, at least as exhibited on my Mac. I receive dramatically fewer emails about iCloud Drive than I used to, which is another data point.

But Mr. Edge Case here can always find a way to break something.2 I’ve been working on a large manuscript for my upcoming book, Flong Time, No See, a revised and expanded collection of researched articles I’ve written about printing and type history over the last several years.3 I pasted in text that was typically copied from Web pages, and wound up with over 250 footnotes.

While putting together the files for this book, Apple released its Creator Studio system, and shipped Pages 15. After having no problems, I dutifully switched over. This was months ago, and as I continued to work on this reasonably large file—weighing it at about 60,000 words now—everything seemed fine.

A few weeks ago, however, I started to experience system slowdowns. Pages was taking up 12 GB of system memory! System load went through the roof! CPU consumption was outlandish! All signs pointed to corespotlightd, a long-time enemy of performance. That daemon handles background indexing for Spotlight, and the slightest thing wrong in a file or directory, or perhaps due to mild corruption in its underlying files, turns it into a rampaging beast that eats processor cycles like I consume potato chips.4

There’s a well-known procedure for killing this beast, though another is spawned from its remains: stop the index, delete the index files, restart, and re-enable indexing. From the Terminal, use these commands. Make a backup before proceeding. Take extreme care with rm -rf, as a misstep in entering or copying can wipe out files!

sudo mdutil -a -i off
rm -rf ~/Library/Metadata/CoreSpotlight

You can use Restart from the Apple Menu while holding down the Option key (to avoid being asked), or you can take a shortcut from Terminal, assuming you have ensured all files are saved:

sudo reboot

After your Mac restarts, go back to the Terminal and re-enable indexing:

sudo mdutil -a -i on

For many (perhaps most) people, this solves the problem. For me, it just delayed it. With some isolation using System Settings: Spotlight: Search Privacy, I added specific folders, such as mail archives, since that seemed like it might be part of the problem.

Search Privacy lets you exclude volumes and directories from Spotlight indexing, which can remove unwanted results and sometimes (only sometimes!) reduce thrashing.

One piece of advice I found suggested that if I thought Pages was the problem, I should exclude the iCloud Drive path containing Pages files. Fine! I clicked the plus icon in Search Privacy, pressed Command-Shift-G to bring up the path dialog, entered ~/Library/Mobile Documents/com~apple~Pages/, and clicked Choose. This seemed to work for a while, then nothing availed.

Finally, after analyzing logs and caches, the culprit was clear. It was not that large a Pages file: every time I saved, Spotlight’s worker bee was performing excessive re-indexing that consumed gigabytes and one or more processor cores. Moving the file to a local directory (in my Documents folder) while editing eliminated the problem. It also freed up space, as the temporary Spotlight indexes and Time Machine snapshots had become absurdly large.

289 footnotes seem to be fine for Pages, but Spotlight and iCloud Drive find them hard to swallow.

What could Apple have done to help here? This is where I would hope machine learning, coupled with some interactive code- and system-expertise bots, could help:

Note that a system component is acting well outside known parameters.

Identify CPU usage that is tied to Apple-owned processes.

Provide a troubleshooting tool, even suggesting it in notifications.

Create human-readable logs that don’t require system administrator knowledge to interpret.

In the case I describe above, an Apple-created daemon was rampaging out of control due to a single identifiable file!—cascading into other problems. It should be solvable by Apple. The fact that we have more power under the hood by orders of magnitude in the past, coupled with neural cores in the CPU packages, just makes this even more embarrassing. Solve Siri, Apple, yes, but real transparency and troubleshooting in macOS would also make a big difference.

[Got a question for the column? You can email glenn@sixcolors.com or use /glenn in our subscriber-only Discord community.]

I should have made an unsinkable/unsyncable pun in the title, but there is an esprit d’escalier moment for everything. ↩

An “edge case” is something that is often considered unlikely to occur with a product, whether software, hardware, or mechanical. Yet, no matter how much I’m just trying to get something done, I apparently trigger edge cases all the time. Software companies “love” me for this. ↩

Flong is a kind of paper-like mold that was extremely useful during the days of metal-based relief printing. Of course, you can pre-order a copy to get the full scoop! ↩

Just ask my family. ↩