By Jason Snell
Siri Recap and Live Rewind debut in watchOS beta
With the release of the first developer beta of watchOS 27.3, Apple is unleashing the last of its Audio Intelligence features on the world: Siri Recap and Live Rewind join Sound Recognition and Music Recognition (a favorite watchOS 27 feature of mine).
These other two new features have been met with a lot of skepticism, as they walk right up to the edge of making your Apple Watch a device for covert recording.
Live Rewind allows an Apple Watch Series 12 or Ultra 4 user to double-click the Digital Crown and get a quick transcript of everything that’s been spoken over the previous 15 seconds. Siri Recap generates AI summaries of every conversation it has been privy to.
Apple, obviously aware that this was going to be an issue, generated an entire Audio Intelligence white paper detailing how the features are secure and private. In general, audio is not recorded (per se), but buffered into a “secure exclave,” a portion of the watch’s chip that cannot be accessed by the system at large, and processed into text. That text is also encrypted and sent to the iPhone that’s paired with the watch, so only that iPhone can receive it and provide more sophisticated AI processing. And any cloud-based models used to generate summaries are part of Apple’s Private Cloud Compute infrastructure, which means they’re end-to-end encrypted and discarded after processing, so nobody (not even Apple) can see the data.
As 27.3 arrives, a few other details have emerged. Siri Recap text, which is based on conversations captured by the user by setting a time frame for capture or manually capturing via Control Center, is deleted after seven days if it’s not used. The user receives notifications on their watch when Siri Recap starts and stops, and a reminder is pushed to the user’s iPhone at the end of the day, pointing at the generated recaps, which are stored in a new Recaps tab in the Siri app.
The Apple Watch itself processes the audio with a “lightweight” ML model that detects if the audio it’s hearing is actually a conversation you’re a part of. (It really doesn’t want to attempt to recap other people’s nearby conversations. If you’re listening to audio on headphones, it takes that as a cue and doesn’t monitor any audio.)
If the audio passes that test, it’s encrypted and transferred to the paired iPhone, which turns it into “condensed text” (some filler words are removed). Then that text is encrypted and passed to Private Cloud Compute, which filters out sensitive personal information and other red-flag content and generates a summary, which is returned to the iPhone for display in that new tab. No content is attributed to specific speakers, and speakers aren’t even identified by name.
Live Rewind is comparatively simple: You double-click the Digital Crown, it makes an audible chime and displays text of the last 15 seconds. That’s done by transferring audio to the paired iPhone, which processes it and returns text to the watch, which displays it for 30 seconds and then throws it away unless the user specifically chooses to save it.
Though these features clearly rely on processing on the iPhone as well as Private Cloud Compute, there’s no option to just record audio on the iPhone and have it generate these summaries. Apple seems to think that a microphone that’s always out in the open (on your wrist) is just a better tool for this kind of feature. Unfortunately, the reliance on the iPhone does mean that if you have two iPhones—perhaps sharing a phone number using iPhone Handoff—only one of them can actually be paired with the Apple Watch, and if that iPhone is out of reach, these features won’t work.
If you appreciate articles like this one, support us by becoming a Six Colors subscriber. Subscribers get access to an exclusive podcast, members-only stories, and a special community.