I'm trying to make a third-party app's on-screen image available to Siri / Apple Intelligence so the user can say something like "send this to " and have the image handed off. And I'd like
to confirm whether I'm using the intended mechanism or whether this particular case just isn't
supported yet.
What I'm trying to do
My app shows a single image (it owns no photo library, the image is an in-app render). I want the
user to be able to reference it as "this" and have Siri move it to another app.
What I implemented (following Making onscreen content available to Siri and Apple Intelligence
and WWDC26 240/343)
A plain AppEntity with a stable id, conforming to Transferable, annotated on the image view with
the iOS 27 .appEntityIdentifier(_:) modifier:
struct OnScreenImageEntity: AppEntity, Transferable {
static let typeDisplayRepresentation = TypeDisplayRepresentation(name: "On-Screen Image")
static let defaultQuery = Query()
let id: String
static var transferRepresentation: some TransferRepresentation {
DataRepresentation(exportedContentType: .plainText) { … } // distinctive text
DataRepresentation(exportedContentType: .png) { … } // the image
}
struct Query: EntityQuery {
func entities(for ids: [String]) async throws -> [OnScreenImageEntity] {
logger.notice("entities(for:) CALLED \(ids)") // <- never fires
return ids.filter { store.isCurrent($0) }.map(OnScreenImageEntity.init)
}
}
}
// on the view:
Image(uiImage: image)
.appEntityIdentifier(EntityIdentifier(for: OnScreenImageEntity.self, identifier: id))
What I observe (iOS 27 Beta 3, device)
"Describe this image" / "Create a note for this" appear to use an automatic screenshot — the
responses reference my app's UI chrome, and EntityQuery.entities(for:) is never called, so my
entity's Transferable is not involved at all.
"Send this to " doesn't attach the image; Siri says something like "I can't attach the
image directly from your screen," which again sounds like a screen grab. entities(for:) is not
called.
The only flow that resolves my entity is the ChatGPT hand-off: asking Siri about the
on-screen content calls entities(for:) (several times). But even there it stalls — Siri responds
"could not clarify what you mean by 'about this'… document, image, or topic?", and no
Transferable representation is ever read (my export closures never run). So the entity resolves,
yet the content is never actually transferred.
The older NSUserActivity.appEntityIdentifier route was never consumed at all; only the iOS 27
.appEntityIdentifier(_:) view modifier produced any resolution, and only in the ChatGPT flow above.
I also understand from WWDC26 240/343 that the cross-app content move is built on
IntentValueRepresentation (entity ↔ a system value type like IntentPerson),
which the recipient imports via IntentValueQuery / IntentValueRepresentation(importing:). Since
there doesn't seem to be a system value type for an arbitrary image (and IntentFile isn't one), I'm
not sure an image entity can participate in that transfer at all.
My questions
Is .appEntityIdentifier(_:) the intended way to expose a single on-screen item in iOS 27? It only
resolves my entity in the ChatGPT hand-off, and even then the Transferable is never read — under
what conditions is the resolved entity's Transferable actually consumed?
Are on-screen content requests ("describe this", "create a note for this") ever meant to use an
app-provided AppEntity + Transferable, or are they always served by screen capture? (For me
they never call entities(for:).)
Is there a supported path for Siri to send an image from a third-party app's on-screen content
to another app? If the transfer requires IntentValueRepresentation and there's no image system
value, is a Transferable image representation ever consumed for this — and if so, how is it
triggered?
If this is simply not supported for image content yet, that's useful to know too, I just want to
make sure I'm not missing a required piece (a schema conformance, an eligibility flag, a different
annotation API, etc.).
I have a minimal sample project that reproduces this (single annotated image, no photo library, logs
to a subsystem so you can see entities(for:) never being called) and filed it as Feedback
FB23813341 — happy to share details.
Thanks!
Topic:
Machine Learning & AI
SubTopic:
Apple Intelligence
Tags:
App Intents
Siri and Voice
Apple Intelligence
Core Transferable
4
3
857