Implementation of Screen Recording permissions for background OCR utility

Question

Created 6d

Replies 4

Boosts 0

Participants 3

I am exploring the development of a utility app that provides real-time feedback to users based on their active screen content (e.g., providing text suggestions for various communication apps). To achieve this, I am looking at using ReplayKit and Broadcast Upload Extensions to process screen frames in the background via OCR. I have a few questions regarding the "Screen Recording" permission and App Store Review:

Permission Clarity: Is it possible to trigger the Screen Recording permission request in a way that clearly communicates the "utility" nature of the app without the system UI making it look like a standard video recording?
Background Persistence: Can a Broadcast Extension reliably stay active in the background while the user switches between other third-party apps (like messaging or social apps) for the purpose of continuous OCR processing?
App Store Guidelines: Are there specific "Privacy & Data Use" guidelines I should be aware of when an app requires persistent screen access to provide text-based suggestions?

I want to ensure the user experience is transparent and that the permission flow feels like a "helper utility" rather than a security risk. Any insights on the best APIs to use for this specific background processing would be appreciated.

Boost

Answer 1

DTS Engineer OP

Apple

5d

Just to confirm, you’re talking about iOS, right?

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

1

Answer 2

tylermclovin OP

3d

yes iOS. Would love to know if this would be possible. Looking forward to your response. Currently there are apps where people take screenshots of conversation from messaging apps feed it to and AI model, and provide text suggestions. But I would live the app to be able to live assist people with text suggestions!

0

Answer 3

DTS Engineer OP

Apple

2d

Thanks for the confirmation. I’ve retagged your thread with ReplayKit because that’s the only option for this sort of thing on iOS [1].

Coming back to your original questions:

Permission Clarity

I don’t think you’ll be able to make progress on this. Broadcast extensions were designed for… well… broadcasting, and the user experience reflects that.

Can a Broadcast Extension reliably stay active in the background … ?

Honestly, I’m not sure. I think this is something worth prototyping before you invest heavily in this project.

App Store Guidelines

I don’t work for App Review and can’t offer any advice on that front. You could ask them here on the forums, over in the App Store Distribution & Marketing > App Review subtopic. However, I think it’d be best for you to create that prototype to make sure you’re your happy with the results before trying to engage with App Review.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

[1] On macOS you have ScreenCaptureKit, which is a very different beast.

0

Answer 4

eddiewangyw OP

2d

One thing worth considering: even if the Broadcast Extension technically works in the background, the UX friction will be significant. Users see the persistent red recording indicator in the status bar, which creates a "surveillance" perception regardless of your actual intent.

For the text suggestion use case, you might want to explore an alternative approach — an accessibility-based solution using the Accessibility API (if targeting macOS) or a keyboard extension that analyzes context within the text field directly (iOS). The keyboard extension route avoids screen capture entirely and might align better with both user expectations and App Review guidelines.

0