Experimenting with Apple’s AI models inside Shortcuts
Of all the features I’m excited about using in macOS 26, the one that most intrigues me is the Use Model action in Shortcuts. Use Model does exactly what you think it does: you toss data into it, and an AI model somewhere (on your Mac, on Apple’s Private Cloud Compute servers, or even at an OpenAI server farm) will take that data and turn it into… something.
The other day, I realized that this new feature would allow me to expand my existing automation that uploads images to the Six Colors web server by adding a description of the image. If there’s something AI stuff is pretty good at, it’s describing images. So I decided to give it a try.
Since Apple’s On-Device model doesn’t support image uploads, I used Private Cloud Compute. Uploading full-sized images to Private Cloud Compute led to very long processing times, so I inserted a step that resizes the image before sending it.
With all that done, the shortcut was able to generate a remarkably accurate description quickly—but it wasn’t quite ready for my use. The descriptions were frequently too long, and they included double quotes that would break if pasted inside the quotes of the alt attribute in an HTML image tag.
A baseball stadium filled with spectators under a clear blue sky. The field is green with a brown dirt infield, and players are positioned on the field. The stands are packed with fans wearing team colors.
No problem. I refined my prompt: “Describe this image for use in the alt text tag on a webpage. Limit yourself to two sentences at most. If it’s a screenshot, please include all the words. Don’t use double quotes, but only single quotes.”
A white dog with black spots standing on a tiled floor in front of wooden cabinets. The dog is wearing a blue collar with a tag. There is a chair with a blue cushion to the right. The floor has a brown and beige pattern.
This changed the output, but did it help? Some of the results were still way too long, and most of them contained double quotes regardless of my commands. I tried a few times to control the length of the result and the use of double quotes, but it was never consistent. Computer programs behave identically every time, but AI does not!
A room with a wooden shelf displaying vintage computers and monitors. A black office chair is in front of the shelf. The wall has framed pictures.
Eventually, I built a shortcut that had to be a hybrid of both approaches. My new prompt is: “Describe this image for use in the alt text tag on a webpage. Limit description to 200 characters maximum. If it’s a screenshot, please include all the words.” The result is frequently still too long, so now my shortcut checks the character count and if it’s too long, it passes the text to Apple’s on-device model with the prompt “Shorten this text to be less than 250 characters.” That works fairly well.
The final step is to use Shortcuts to search for the double-quote symbol and replace it with single quotes, because I really can’t rely on the AI to pull that one off. As with so much AI stuff, it’s simultaneously an amazing piece of technology and incredibly stupid.
And then there’s the final step: human intervention! My automation is meant to generate the HTML I want to insert in my stories, but that means I always see it and can adjust it as needed. There’s always a human being looking at that text and making sure it makes sense, which is good, because there will be errors. I got one description that inexplicably added on hashtags(!) and another that helpfully included the current time in a description of an Apple Watch screenshot—only to get the time entirely wrong.
Adding these kinds of tools to Shortcuts is going to be a real learning curve for all of us. And I’m sure app developers are feeling the same way about their new access to Apple’s on-device models. AI is a tool that can do some amazing things—but programmers (and Shortcuts creators) will still need to apply adult supervision.—Jason Snell
Handling expenses with AI: But at what cost?
Like Jason, I’ve been playing around with the Use Model action in Shortcuts. For me, the biggest attraction was the idea of conquering a workflow that I’d never been able to nail down before: filing my expense receipts.
You know me, I like a good expense tracking spreadsheet, but one of the pain points in my workflow is getting the receipt (often from my email), turning into a PDF (if it isn’t already), then laboriously entering the details into my spreadsheet. In the past, I’d attempted to create a shortcut to simplify entering this information, but the only real way to isolate the data I wanted was to enter it manually via a succession of prompts for the vendor, amount, and so on. Not exactly a timesaver from tabbing through spreadsheet columns.
But pulling information out of a document—especially information that might appear anywhere in a variety of forms—seems like something an AI model would be good at, so I decided to take another crack at it with Shortcuts’s new AI capabilities.
I started out my workflow by grabbing all the text from a PDF or web page, then passing it to the Private Cloud Compute model. (I attempted to use the On-Device model at first, but it was both very slow and not quite as good at formatting the response in the manner I wanted.)
At first, my goal was to get comma-separated values for the date, vendor, and dollar amount that could then be passed into my spreadsheet, but when that didn’t quite work I tried both tab- and newline-delimited and sent them to the clipboard, then tried to paste the values into my Numbers sheet. Turns out that doesn’t work great: Numbers really dislikes pasting things into multiple columns.
But after doing a bit of digging, I discovered the ideal format to pass to the “Add Row to Numbers Spreadsheet” action is actually a list. So I told the model to pass back my values as comma-separated values, then used the Split Text action to separate them on the commas, which automatically generates a list. At that point, I could simply pass that list to the Numbers action, and the values would get put in the first three rows of my sheet.1
The prompt I settled upon—after a lot of tweaking—was this:
This information is a receipt for payment. Please return the amount of the expense in US dollars, the date of the expense (look for dates in email headers if it’s formatted as such, otherwise use the curent date), and who the vendor was (use the subject line and from line in email headers if formatted as such). Also, create a new filename in the format: YYYY-MM-DD-[first five alphanumeric characters of vendor in uppercase]-[full dollar amount with no decimal point or dollar sign]
Format the values as follows, separated by commas:
date in MM/DD/YYYY,vendor,full dollar amount (make sure to include any decimal point but not the string USD),filename
Okay, so you probably noticed I put a whole separate section in there about a filename. That’s because I had the bright idea to see if the shortcut could rename the original file I gave it my receipt format.2
However, in typical AI fashion, this has introduced some problems. For one thing, the AI model gets confused that I’m asking for the dollar amount in two different formats. While I want the spreadsheet value for to be normally formatted—say $209.49—while I want the filename version to simply be the digits 20949. Unfortunately, sometimes it dumps decimal-less version into the spreadsheet, yielding a heart-stopping surprise expense of $20,949.
Another time-honored AI problem: counting. Oh, if these machines could count to five, perhaps they could rule the world. When I ask it to use the first five alphanumeric characters of the vendor, it first gave me six characters. Sometimes it will give me five…but not all the time.
And therein lies the rub with all of this. The results are neither reliable nor necessarily repeatable. The same data run through this shortcut multiple times provides different answers: I’d think that anathema (not to mention madness inducing) to the sensibilities of any programmer. Given the same data, the algorithm should yield the same thing every time, but the non-deterministic nature of AI models throws that out the window. Perhaps I could engineer an even better prompt that would get closer, but unlike other programming tasks I’m not even sure that I could test this enough to feel confident. I could run it 99 times and the 100th it could spit back something different and totally incorrect.
Given this, I find it no wonder that Apple punted on some of its promised Apple Intelligence features from last year. What I’m attempting is a relatively simple task for an AI model and it still gives me the wrong information sometimes. How confident would you be if it tells you when your mom’s flight lands?
I seem to have had some success in tweaking the prompt by explicitly asking for separate dollar amounts, but I’m still hardly filled with confidence. What I’ve ended up with is essentially a not very bright assistant whose work I need to double check. And if I can’t trust that work, does it actually save me time?—Dan Moren
- Perhaps my favorite discovery of writing this workflow: for reasons that will shortly become apparent, I ended up with a list of four items, but wanted to pass only the first three to Numbers. However, there’s no facility in Shortcuts—that I could find—to remove a list item. Until, that is, I stumbled across a fascinating workaround. The “Filter Files” action, as its name suggests, is intended for files, but it actually works on any list you pass it. So I gave it my list of four values and told it to limit it to three items. Worked perfectly. ↩
- I originally tried doing the filename separately with the On-Device model, but again, it proved unreliable. ↩
If you appreciate articles like this one, support us by becoming a Six Colors subscriber. Subscribers get access to an exclusive podcast, members-only stories, and a special community.