By Joe Rosensteel
Casting an expert eye on Apple’s AI photo-retouching features
In its 27 operating system releases, Apple added loads of generative image editing tools, and the tools have incrementally changed since they were introduced over the summer. I’m going to take my 20-plus years of experience in visual effects and talk about how well those tools work in practice.
I’m not going to get into a whole preamble about how I really do think it’s OK for you to edit a photo, or the video comparing the initial release of Clean Up with comparable retouching tools. People can do bad things with image editing tools, but I don’t believe that photos people take of their lives should be treated like courtroom evidence.
Having said that, it’s important that tools set users up for success by guiding them toward making good choices that preserve as much of their original images, and their original intent, as possible. Data loss of original pixels can be a trade-off to get an adjustment that people want, but tools shouldn’t hide the data loss from users.
Lesson number one in VFX is that you’re only changing the pixels you have to change for the effect the client wants. You do not de-noise (or de-grain) your entire image (or image sequence) to do your work and then send that back to the client. You do not re-noise the entire de-noised image either; if it can be helped, you control the region you are editing in your image to confine your work. As much of the original image as possible should pass through.

Apple doesn’t always disclose to the user how its generative tools are impacting the images. That includes things like maximum resolution caps, filtering the pixels that are being kept, blurry boundaries, and potential color and value shifts. There is the Show Photo Without Edits button, which Apple provides to let people see before and after a generative edit has been made. I strongly encourage everyone to use that button. But it won’t tell you about lost resolution, and when the image is scaled to fit to the window, you may not be able to see the damage.
It’s totally fine if you’re happy with the zoomed-out version, and it suits the purpose (a lock screen image, a desktop, whatever). For a lot of people, they’re the same picture. Just know that these tools don’t really match your original sensor noise, and as of the 27.2 release the images are now watermarked with SynthID, even though Apple does not disclose that to users in the interface, so they can’t decide if the change they’re making is worth the image being “AI” since SynthID doesn’t disclose the editing chain.1
Pretend extend doesn’t just append
Cutting off the edges of a photo, and adding on to the edges of a photo, are unsurprisingly related. So much so that Apple reused its aspect-ratio crop selection code for the interface of its Extend tool. (This should be in the Crop interface.)
At the very least, Apple should change the code it’s reusing from the aspect ratio bounds resizing so that it fits to the bounds of the input as a minimum dimension when toggling between 3:4 and 9:16. Because right now, changing the aspect ratio in Extend can crop the image, which is what you want for Crop… but not for Extend.
I don’t like the hazy blur effect at the edges, because users should be able to manipulate a single edge of their image without messing with pixels all the way around. There’s no useful way to snap to fit the original bounds in order to use as much original image as possible. You just have to kind of eyeball where you’re dragging the reticle.
The unfortunate truth is that Extend throws away image data. It only looks like it keeps your original pixels when you’re zoomed way out. It perceptually reads like you’ve extended your image, but what’s really happened is that the original image is actually scaled down and resampled to be a smaller resolution so that your generative border, plus your original pixels, all fit in a 12.19-megapixel canvas.
It appears that Apple is using a soft filter to resample the image—not even a sharpening filter, because everything is definitely softer and there are no ringing artifacts associated with sharpening.
The blurry border from the crop preview is also true to what’s happening, because the entire border is blurry. There seems to be about 200-300 pixels of a feathered mask operation that takes your scaled-down photo and combines it with the generative edge.



Imagine that I said that I wanted to add a room to a one-story house that had no property boundary constraints, and the contractor did it by demolishing the house and building a scaled-down replica of the house, then adding the extra room on so the whole house took the same square footage. That’s some Nathan Fielder behavior.
If the generative maximum is 12.19 MP, or something around that, why not take those generative pixels and scale those up to match the increased bounds so that the original core can be preserved? Don’t blend any borders that aren’t actually being extended! Forcing everything to fit some technical constraint rather than making preservation of data the constraint is backwards.
Another place where data could be pulled from to improve quality is other neighboring photos you’ve taken. Extend doesn’t use other photos in your camera roll from the same time frame, or location, as context. You can easily see this for yourself if you duplicate a photo, crop the photo, and then use Extend on the crop, you won’t get buildings sourced from the neighboring image. It’s a silly test, but it just confirms it’s not happening.
Context is very important to getting good results, because it’s what a real photo retoucher would do, or a VFX artist. It’s one thing if you don’t have the data, in which case filling in what was probably there is much more impressive, but if you could have filled in what you have photos of, then you should do that.
Which brings us to…
Reframe to blame for eye strain
The Reframe feature is the most “we repurposed something” of the models. I would argue that it is not a creative image editing tool at all, but it is very clever. If you look at Apple’s ML Sharp, or any of its other myriad projects around 3D computer vision, you can see a lineage of sorts. (This means that things like reflections and refractions do weird stuff. Pond surfaces, shop windows, wine glasses, and the like.)
Reframe lets you move around quite a bit, and you’ll see all the stretchy weird stuff, and holes, from areas where the camera did not have line of sight. Reframe does not use context from surrounding photos, or video, to fill in those missing pieces, unfortunately. Instead those holes are filled in with speculation from some kind of model.
Reframe doesn’t retain any of your original images, even scaled down, like Extend does. It’s filling in everything seemingly guided by its extrapolated 3D view that’s full of holes. It doesn’t have access to the original imagery to paste back in.
However, where Extend mostly softens and mushes original pixels, Reframe also has tone and color shifts that are not anchored to your original image.



These color and value shifts are not uniform, and aren’t restricted to certain file types. The SDR image from my Sony a6400 has shifts, and the egret photo was taken with Project Indigo, not the Camera app. There isn’t an entirely consistent and predictable direction of color and tone shifts. Sometimes it’s more drastic than others, or a defect might be similar with shots taken around the same time of day.
To take the angle adjustment out of the calculation I took a bunch of shots with gray and white to measure color cast changes and only changed the image by 0.99 zoom. This artificially small adjustment also shows how much Reframe hammers the image since all pixels need to be redrawn, but it’s much easier to compare directly to source than if I was wildly swinging the camera around. If you want to do your own tests I would recommend a tiny adjustment like this to make comparisons easier.



This is technically very sophisticated, and the part where you can just drag your mouse, or finger, around to move things is pretty neat. Unfortunately, my critique of the control scheme is that people can’t quite do the kinds of recomposition that they might want to do to a photo from a 2D perspective. “Move this person to the left” has to be “Move the camera around this person” instead.


Here’s my partner Jason in front of Tower Bridge in London. I like the photo just fine, no regrets, but it’s arguable that it would have been helped if Jason was standing slightly to screen right. These controls can’t move Jason. If I shift the perspective until Jason’s body is no longer overlapping with Tower Bridge then Jason is no longer facing the camera, and we’re looking at him from a three-quarter view instead of front on.
I struggle to think of a use case for orbiting around a person unless you’re Michael Bay.
In another example, where I took a selfie of Jason and me at the Eiffel Tower, I tried to use Reframe to make it look like less of a selfie (again, I don’t care that it looks like a selfie, but I know normal people will do these things). I’m not swinging wildly off axis, but more of the background is revealed from parallax, and not sourced from context. Most importantly, Reframe caught my knuckles and turned them into some kind of cotton candy cancer.


Big nope. Surely, I can use Clean Up to remove my knuckles?
Clean Up re-up
The new Clean Up is not what I would consider to be two years better than the original Clean Up. It couldn’t paint my knuckles out without inserting a flesh-colored seam in the shoulder, or other streak artifacts. I don’t understand why this is quite so bad at simple requests.
But it turns out that there are two Clean Ups: when the Image Quality is set to Fast and when it’s set to High Quality. (There’s also an “Auto” setting that usually gave me Fast results.) The High Quality pass involves using a server-based model, so it’s slower… but it’s also much better.
A flaw in both Clean Ups is that there’s image segmentation that tries to intelligently only select the hand I want to remove, which always leaves a few pink pixels where the hand pixels are at their thinnest just as they disappear over the shoulder edge. I had to hit that with another round of paint. This isn’t a model limitation, like when Fast Clean Up added pink pixels; this is a selection issue.
The good news is that Clean Up is the only generative tool in Photos that doesn’t alter the entire image. It does seem to dilate out from the paint operation and blend back to the photo, so the trees above my hand shift, but that’s more in line with what I would expect from this tool than what Extend and Reframe do.
It would be nice if I could more precisely paint where I wanted it to work, not where I wanted it to selectively decide to work, though.
Crop shop image plops
The concerns I have about unnecessary modification to original pixels are not the kinds of things that everyone in the general public would notice or object to. It might look a little shifted, but they just wanted this for some lock screen, photo frame, or highly compressed Instagram carousel. I do think it’s up to Apple to do its best since there is no technical reason for Apple not to.
The real obstacle for most people will be tool use.
Providing these individualized models to people that have some overlapping function exposes the models Apple has ready to ship, not the workflow it sees people using. (The most basic example being the need to provide a one-stop shop for cropping and composition tools.)
I would also remind Apple that it doesn’t ship a competitive set of tools for boring, deterministic, 2D operations like straightening, perspective operations, and subject-aware cropping. How many times have we collectively clicked the “auto” button in the Crop panel and it didn’t do what we wanted?
Why are we fighting with center-weighted vertical and horizontal perspective tilt operations instead of having an automated perspective adjustment like Lightroom? Instead, we have an assortment of nondeterministic image diffusion models running on servers, which is way more complicated. We can’t just hit a button to fix some skewing lines with a simple 2D operation? Something that doesn’t put the scarlet letters “AI” on the image?
I get it that generative models sound fancy, but this is for a public that is not going to scrutinize the tools as much as image-editing nerds are. The tools aren’t good enough just because normal people can’t tell how much of their original image was destroyed. I know that Apple wants these tools to seem magical, but they’re all still kind of just demos that have escaped containment. I’m not sure they solve as many real-world problems as Apple would like them to.
- Provenance is really important and I’m glad that Apple added a way to identify synthetic images, but it doesn’t disclose that to users. A support document says that users can verify SynthID at synthid.com, but there’s nothing in the interface that communicates this. ↩
[Joe Rosensteel is a VFX artist and writer based in Los Angeles.]
If you appreciate articles like this one, support us by becoming a Six Colors subscriber. Subscribers get access to an exclusive podcast, members-only stories, and a special community.