Apple’s Camera Chief Thinks AI Can Give You Superpowers

Comment
LoaderSave StorySave this story
Comment
LoaderSave StorySave this story

What even is a photograph these days?

As tech giants pack generative AI capabilities into our phones and their camera software, the line between what is a real image and what isn't continues to blur. Phones from Google and Samsung, for example, now come with features that let you drastically alter a photo by erasing people, moving people around in the shot, and even adding new objects to the scene.

Apple is getting in on the action by adding new generative features to its Photos app, though the company's iPhone camera chief, Jon McCormack, stresses that Apple is taking a more measured approach than its competitors and isn't “doing AI for the sake of AI.”

At its annual Worldwide Developer Conference on Monday, Apple showed off a handful of AI features invading the Photos app in iOS 27, which will arrive on iPhones later this year.

While the iPhone’s Photos app already has the Clean Up tool, which lets you erase unwanted objects in pictures, it'll perform even better in iOS 27 thanks to its access to Apple's improved AI models. However, there are two new features—called Extend and Spatial Reframe—that let you expand the space around your photo or change the perspective of an image, all while generating fake pixels. The camera “thinks” about what should be there, then draws it in.

McCormack says there's a giant backlog of unsolvable issues that AI is now helping to address and that these new features are very deliberate. “You don't have to know all the details of how to do something in Photoshop or something else—it gives normal people these absolute superpowers,” McCormack says.

Apple doesn't want to let you run wild with your images and generate all kinds of fakery, though. (At least not in the Photos app; the App Store offers plenty of tools for making photorealistic slop.) The fake pixels the Photos app generates are restricted to what's in the background. It won't alter the pixels of the main subject's face. With Clean Up, for example, you cannot remove the primary subject in the image. The Extend function only works once and expands the image by 25 percent—you can’t save, edit the image again, and infinitely extend it with AI.

McCormack also says Apple will integrate Google DeepMind's SynthID technology later this year to add an invisible watermark indicating these images have been altered with generative AI. Any platforms where you share the photo may be able to flag it as AI-edited. (Just know that researchers have shown that digital watermarks aren't foolproof.)

“A photograph is of something that actually happened,” McCormack says. “We really do believe in this idea of authentic journalism to your own life—when you're capturing photographs, you're making these memories, you're putting moments of your life in a bottle that you can go back to. It's really important to us that we create tools that keep the sanctity of that moment.”

The head of Google's Pixel camera division shared a similar sentiment when I spoke to him a few years ago; however, he highlighted the importance of how you remembered the photograph. Google is far more lax in letting you alter the image to match what you had in your mind. Was the sky bluer in your memory? Go ahead and change it.

Apple's new tools are more restrictive and are specifically meant to solve compositional problems you may not have realized you made during the capture process, McCormack says. Maybe you didn't see that unsightly plastic bag rolling through the background. Maybe you took a picture of your kid but shot a little too high up. Or maybe you framed your spouse too close to the edge and want a little more space.

Della Huff, product manager for Apple's Camera and Photos software, says the team spent time training the AI models to minimize any hallucinations that might occur during these digital adjustments. “It's not going to create anything that shouldn't be there," Huff says. For example, if you're trying to extend a street scene, it's plausible that there could be a car parked outside the boundary of your original photo, but Extend won't make that assumption and generate it. “How the model has been trained is if you don't need to create something there, then just don't—do the minimum amount of hallucination to achieve the goal the user is asking the model to do.”

But in one of my examples of using Extend in the iOS 27 developer beta, I took a photo of a friend sitting at a table, then tried to expand the scene a little more to the right. In the background, some people were sitting at tables, but as the scene was extended, the Photos app added a couple more tables, complete with fake people—people who were never there in real life—sitting at them.

Huff says the feature is trying to match the existing aesthetic. If there are people in the background but Extend doesn't add anyone else as you expand the photo, that may look strange. “If we said the rule is we could never generate a background human ever, then the feature would become less useful,” she says.

It's worth remembering a point Apple made at the keynote: There are usage limits for these new camera features. Apple isn't sharing exactly what those daily limits are, but users will need to subscribe to iCloud if they want to Extend, Spatial Reframe, or Clean Up their photos multiple times a day.

One of the big themes of Apple's WWDC presentation this year was using natural language to get things done. You can converse with Siri naturally—no need to use rigid commands—and it will understand your intent. In the Calendar app, you can describe the event, and it will create it in a jiffy, no need to fill out various fields. You can describe the shortcut you want to create in the Shortcuts app without fiddling with triggers and actions. In Safari, you can speak an extension into existence. So how come you can't edit a photo with natural language? That's a feature Google introduced last year on Google Photos.

Huff says the new Siri AI can handle a few touch-ups on your behalf and that Apple is not ruling out anything on the future road map. Siri cannot edit with the new AI features—those are strictly human-controlled—but this is partially because Apple doesn't think it’s a good user experience. Those features require a little more muscle; it’s hard to talk Siri through the process of changing a photo's perspective.

“There’s so much more that’s open-ended about that, especially something like Spatial Reframing where that’s really a user-intent thing that you need to express,” Huff says.

Speaking of Siri, the other big change coming in iOS 27 is that Siri is getting injected into the Camera app. McCormack says Sirifying the iPhone’s camera is purely about reducing friction.

Siri’s Visual Intelligence feature—which operates like Google Lens by using computer vision to study an image—currently is activated through the Camera Control button. But this is a camera-specific function, so McCormack says it makes sense that it would live in the Camera app.

“It’s embracing the idea that the camera is really a number of things,” he says. “It’s a memorialization device … it’s a note-taking device … or I’m just curious what that plant is.”

Apple's more measured approach might feel at odds next to its image-generation app—Image Playground—which lets you create AI-generated images with text prompts (or by injecting a photo from your own library). In iOS 27, Image Playground defaults to generating more photorealistic pictures unless you specify a particular art style. But McCormack says these different uses of AI will feel different based on the context of where they live on your phone. If you're in the Photos app, users want to know they're in a safe place where memories are kept intact. But in Playground, the naming is intentional—“it's a place to play.”

“Both use cases are totally valid,” Huff says. “I want to improve this photo. A photo is something that happened, I captured it with my camera, and the Photos app is where I can improve it. But I also want to be creative and let my imagination be able to run wild, and so they’re two separate experiences intentionally.”