Posts under Machine Learning & AI topic

Post

Replies

Boosts

Views

Activity

AppIntent ignores registered dependencies when awaited
App intent has a perform method that is async and can throw an error, but I can't find a way to actually await the result and catch the error if needed. If I convert this working but non-waiting, non-catching code: Button("Go", intent: MyIntent()) to this (so I can control awaiting and error handling): Button("Go") { Task { do { try await MyIntent().perform() // 👈 } catch { print(error) } } } It crashes: AppDependency with key "foo" of type Bar.Type was not initialized prior to access. Dependency values can only be accessed inside of the intent perform flow and within types conforming to _SupportsAppDependencies unless the value of the dependency is manually set prior to access. Although it is invalid since the first version is working like a charm and dependencies are registered in the @main App init method and it is in the perform flow. So how can we await the result of the AppIntent and handle the errors if needed in the app? Should I re-invent the Dependency mechanism?
1
0
1.1k
Aug ’26
MLXVLM factory fallback doesn't catch a vision_config-present-but-no-tower-weights checkpoint (Qwen3.5-9B)
I ran into a gap in the MLXVLM → MLXLLM factory fallback and wanted to check whether this is expected behavior or something worth filing. Context: I maintain a Swift-native macOS agent built on MLX Swift, with a hardware-adaptive catalog of local models. I tried adding mlx-community/Qwen3.5-9B-4bit as a plain text model. Its config.json ships a full vision_config block — it looks like a multimodal config — but the actual checkpoint has no vision tower weights at all. It's genuinely text-only; the metadata just doesn't reflect that. ModelFactoryRegistry tries MLXVLM before falling back to MLXLLM. I expected that to fail cleanly and fall through, since there's no vision encoder to load. It doesn't. MLXVLM's Qwen35Configuration declares visionConfiguration as a non-optional property mapped to vision_config. Since the key is present in the raw config, decoding succeeds — there's nothing to catch at that stage. So MLXVLM proceeds to actually load the model as a VLM, and it crashes later, inside WiredMemoryUtils.tune(), because getRopeIndex() expects real vision-encoder dimensions to compute against, and there are none. Not a clean decode-time error I could catch — a runtime crash several layers past config parsing, in the RoPE indexing math. The workaround I ended up with is config-level surgery before the model reaches the factory at all: strip vision_config (and image_token_id / video_token_id) from the JSON first. With the key genuinely absent, Qwen35Configuration's decode throws DecodingError.keyNotFound as the fallback logic presumably intends, MLXVLM fails cleanly at that stage, and ModelFactoryRegistry falls through to MLXLLM correctly. That works, but it feels like a workaround rather than the intended path — the factory fallback seems to assume any failure will happen at decode time, and a config key that's present but semantically empty (no real vision weights behind it) skips that safety net entirely. Questions: Is there a more canonical/official way to detect whether a checkpoint actually has real vision tower weights, short of opening model.safetensors.index.json and counting vision_tower.* tensor names myself? (I've done exactly that for a different, genuinely multimodal Qwen VL checkpoint — 333 real vision tensors there vs. zero here — but it feels like something the loading path should be able to tell me.) Is the vision_config-present-but-empty pattern something the mlx-community conversion pipeline is aware of, or is this specific to how Qwen3.5-9B-4bit happened to get converted? Happy to share more logs/repro details if useful.
0
0
409
Aug ’26
How can a local AI agent use MLX/Metal unattended on macOS while remaining confined to an authorized workspace?
How can a local AI agent use MLX/Metal unattended while remaining confined to an authorized workspace? I am developing an AI-driven local media-processing workflow on an Apple-silicon Mac and am trying to understand the correct architecture for allowing it to run unattended without giving the AI agent unrestricted access to my primary personal computer. I am not a software engineer, so I may be missing an established macOS mechanism or using the wrong terminology. I would appreciate guidance from people familiar with MLX, Metal, sandboxing, and macOS security. What I am building I use OpenAI Codex as the local execution/software-development agent. The working system currently: ingests and verifies original video and still media while preserving immutable originals; performs visual semantic analysis and divides video into meaningful time-coded segments; separately analyzes spoken language rather than assuming audio and video are semantically equivalent; uses MLX Whisper locally on Apple silicon for time-coded speech transcription; stores visual and language semantics in a relational SQLite media catalog. These five stages are working. My current test corpus contains 148 original media files, 126 visual semantic segments, and 765 speech segments. The next stages are AI editorial construction from the semantic database and generation of instructions/scripts for a DaVinci Resolve rough cut. The security architecture I want Codex to operate autonomously within a deliberately bounded development environment. I do not want to solve this simply by granting an autonomous agent Full Disk Access to my primary personal Mac. The concern is ordinary fault containment. Codex generates and executes scripts, invokes applications and command-line tools, and manipulates files. A mistaken path or defective generated script should not have unrestricted consequences for the rest of my computer. I therefore separated AI execution from ordinary personal files. Codex is configured for Workspace Write access with explicitly authorized project roots. Canonical media resides on a separately authorized external SSD, and temporary AI working artifacts are kept separately. Ordinary Python and FFmpeg operations now run autonomously within these authorized areas. The problem The difficulty appears when the workflow invokes capabilities that cannot operate inside the ordinary Codex sandbox. The clearest example is MLX Whisper. I am using: MLX Whisper 0.4.3 mlx-community/whisper-small-mlx Apple silicon local transcription MLX Whisper works successfully and its transcription quality is sufficient for my semantic-retrieval application. However, MLX could not access Apple Metal/GPU execution from inside the ordinary Codex sandbox. Codex therefore requested permission to execute the transcription operation outside the sandbox. Once approved, MLX/Metal worked and the entire corpus was successfully transcribed. The processing therefore works, but the workflow cannot run genuinely unattended. A future operation should be able to run: new media → integrity verification → visual semantic analysis → MLX Whisper transcription → language semantic analysis → SQLite update → QA But if execution stops midway waiting for a human to click Allow, the pipeline is not operationally autonomous. What I have already tried I initially encountered permission problems even with ordinary file operations. I therefore: separated Codex work from ordinary personal documents; created dedicated project/work areas; explicitly authorized the required working roots; configured Workspace Write; separately authorized the external media repository; tested shell/Python and FFmpeg operations within those boundaries. Those changes worked. Routine Python and FFmpeg operations now run without approval prompts. The remaining issue occurs with MLX/Metal and some other application/runtime operations that require sandbox escalation. My question Is there a supported architecture for allowing a local AI agent to invoke MLX/Metal and other deliberately authorized development tools unattended, while still confining the agent to defined project/workspace boundaries rather than granting unrestricted access to the entire Mac? For example, should I be investigating: App Sandbox entitlements; a signed helper tool or XPC service; security-scoped resources; a dedicated executable with appropriate entitlements; a different method of launching MLX/Metal; or another macOS mechanism? In particular, can Metal/GPU access coexist with persistent bounded filesystem access without requiring interactive approval each time the AI invokes it? I am also unsure which security layer is actually responsible here: the Codex sandbox, macOS App Sandbox, TCC, executable/code-signing rules, Metal restrictions, or some interaction among them. If this kind of bounded unattended execution is intentionally not supported, that would also be useful to know. My alternative would be a dedicated Apple-silicon Mac containing only the AI-development environment and replaceable project data, where broader permissions would have a much smaller failure domain. I can provide the Codex configuration, exact successful and failing commands, directory/root configuration, macOS/hardware information, and sandbox diagnostics. I would particularly appreciate guidance on which security layer is causing the MLX/Metal escalation and what the supported architecture would be for this use case. Thank you.
0
0
899
Aug ’26
Enhancement: pre-action policy hook before multi-step App Intent / Siri AI perform()
As App Intents power multi-step workflows via Siri, Shortcuts, Spotlight, and Apple Intelligence, I would like to request a platform pre-action policy surface: Before intent perform(): input: intent_id, parameters, caller_agent_id, session_id, risk_hints output: allow | warn | deny | require_confirmation side: local audit_receipt Why this is needed Sandbox and TCC solve app isolation and one-time permission grants. Multi-step agentic loops introduce a different risk shape: Destructive tool selection from noisy natural language2. Uncapped retry / network loops3. Unauthorized data movement across app boundaries4. Financial / identity / Wallet actions that need dual control even when the app is already authorized Prompt-only or documentation-only guidance is not enforceable mid-loop. Apple-aligned properties On-device evaluation by default (privacy)- Complements TCC / sandbox â does not replace them- User-visible WARN / DENY with recoverable explanation- Optional Instruments-style â agent action timelineâ with gate verdicts Risk classes (map to entitlements mental model) | Class | Example | Default posture || --- | --- | --- || Read local | calendar, on-screen text | Allow / low friction || Cross-app write | paste into finance app | Warn or confirm || Destructive | delete, wipe, revoke | Deny or hard confirm || Network exfil | send sensitive context off-device | Deny until confirm || Financial / Wallet | transfer, mint pass | Always confirm | Related discussion There is already an excellent thread on pre-effectuation / execution finality for high-consequence App Intents. This request is complementary: a first-class, developer-visible pre-action hook (ALLOW / WARN / DENY / CONFIRM) at the intentâ action boundary, plus local audit receipts. Shipping reference (independent OSS control plane) I ship ThumbGate (npm thumbgate) â a local-first pre-action firewall used today for AI coding agents: PreToolUse / MCP interception, ALLOW/WARN/DENY, thumbs feedback â prevention rules. Public: https://thumbgate.ai · https://github.com/IgorGanapolsky/ThumbGate I will also file this as a Feedback Assistant Suggestion. Looking for: Confirmation this belongs under App Intents / Apple Intelligence2. Any existing API I missed for third-party or system-level pre-perform gates3. The right internal owner / Feedback area if this should go through a different channel Happy to share a one-pager with the risk taxonomy and proposed hook shape.
0
0
607
Aug ’26
iPadOS 27 Beta — Siri AI overlay causes no Scene lifecycle callbacks, starves BT data processing threads
Environment: iPadOS 27 Beta (Developer Beta) iPad with Bluetooth Classic (iAP2/ExternalAccessory) + BLE active session App uses UIKit, WKWebView, scene-based lifecycle Problem: When the user invokes the new Siri AI by long-pressing the power button while our app is in the foreground with an active Bluetooth Classic session, we observe: No scene lifecycle callbacks fire — no sceneWillResignActive, no sceneDidEnterBackground, nothing. We confirmed by logging every UISceneDelegate method. Main thread / data processing threads are starved for ~2 seconds, causing a backlog of incoming Bluetooth data. Our real-time data processing latency jumps from ~105ms to over 2,300ms within 2 seconds of Siri activation. CADisplayLink / requestAnimationFrame callbacks show a ~935ms gap coinciding with the Siri overlay appearance, then irregular intervals afterward. The Bluetooth Classic transport (ExternalAccessory/iAP2) remains physically connected throughout — the issue is purely host-side processing starvation. What we've ruled out: BLE link degradation: firmware-side diagnostics confirm 100% data delivery, 0 lost packets during the incident Memory pressure from our app: our process memory stays flat; system-available memory drops ~14 units externally Questions: Is the absence of sceneWillResignActive when Siri AI activates on iPadOS 27 intended behavior, or a beta bug? The new UIApplication.systemPrefersReducedResourceUsage property (iPadOS 27 beta) — is this intended to signal system overlays like Siri consuming resources? Does the corresponding systemPrefersReducedResourceUsageDidChangeNotification fire when Siri activates? Are there recommended patterns for apps with real-time Bluetooth data processing to maintain thread priority during system overlays? We currently use default QoS for our data processing dispatch queues. The processing starvation causes the waveform display to degrade (appears as a connectivity issue to the clinician) even though the wireless link is healthy. We need either: A notification that a system overlay is active, so we can adjust our UI accordingly Guidance on maintaining processing priority during Siri AI activation Any community insight on workarounds would be highly appreciated. Thanks.
2
0
1.5k
Aug ’26
The new beta Siri disappeared
I've already used the new Siri for a few weeks but now it disappeared. The Siri app still exists but cannot use. Currently the Siri is the original Apple Intelligence. I went to Settings, Screen Time, Content & Privacy Restrictions, Siri, and the Siri AI (Beta) is grey cannot choose. Anyone knows how to fix it?
0
1
370
Aug ’26
Pre-Effectuation Execution Finality for Siri and App Intents
I would like to discuss a possible security architecture for Siri, Apple Intelligence, and App Intents where permission to invoke an app action is separated from permission for that specific action to become externally effective. For low-risk actions, existing authorization may be sufficient. However, for higher-consequence actions—such as payments, file export, message transmission, account changes, device control, or other irreversible operations—there may be value in introducing an additional execution-finality boundary. Problem Space An AI assistant may be authorized to invoke an App Intent, but that does not necessarily mean every resulting action should immediately become effective. For example: Siri may be allowed to invoke a payment-related intent, but not every amount or recipient should necessarily be executable. An app may expose a file-sharing intent, but a particular file or destination may fall outside the permitted scope. An AI-generated message may be validly created, but its final transmission may require additional execution-specific validation. Context, permission, destination, revocation state, or security state may change between intent generation and actual execution. The proposed distinction is: Permission to invoke an App Intent versus Permission for the specific resulting act to become externally effective Proposed Architecture A high-consequence action could first become a Candidate Act and remain in a Non-Effective State. Conceptually: Siri / Apple Intelligence ↓ App Intent ↓ Candidate Act ↓ Non-Effective State ↓ Protected Validation ↓ Scoped Execution Authority ↓ Finality Sink Verification ↓ External Effect If validation fails, expires, is revoked, is replayed, or becomes ambiguous: Default Denial → No External Effect Short Definitions Candidate Act The specific operation proposed by Siri, Apple Intelligence, or an app before it is allowed to create an external consequence. Non-Effective State A state in which the operation may be prepared, inspected, or evaluated but cannot yet produce its intended external effect. Protected Validation A validation step checking execution-relevant conditions such as app identity, user authorization, purpose, destination, scope, freshness, limits, revocation state, or device security state. Scoped Execution Authority Authority limited to the specific validated action rather than a broadly reusable permission. Finality Sink The consequence boundary where the action becomes externally effective—for example, a network transmission, file release, payment commit, database change, or physical-device action. Why This Could Be Useful This architecture could provide: separation of AI decision-making from final execution authority; fail-closed behavior for invalid or uncertain actions; resistance to replay or stale authorization; action-specific rather than broadly reusable authority; validation closer to the actual consequence boundary; stronger control for increasingly autonomous AI workflows. This would be intended as an optional mechanism for higher-consequence actions, not as a replacement for App Intents, existing user authorization, entitlements, sandboxing, or other Apple security mechanisms. Possible Interoperability Relevance This question may also become relevant as operating-system interoperability requirements evolve, including in the European Union under the Digital Markets Act. As third-party AI assistants and services gain deeper interoperability with operating-system features, there may be a need to distinguish between allowing an interoperating service to request an action and allowing that specific action to cross the final consequence boundary. A device-side execution-finality mechanism could potentially provide a technical middle layer: third-party AI services could request interoperable actions, while the operating system retains a neutral protected mechanism for validating the specific action immediately before it becomes externally effective. This may help explore how broader interoperability and strong device-side security could coexist without requiring unrestricted execution authority for either first-party or third-party AI assistants. I would be interested in whether Apple considers this type of consequence-boundary enforcement compatible with existing or future App Intents and interoperability architectures. Questions for Apple Engineers and Developers Does App Intents currently provide a supported mechanism for maintaining an action in a non-effective state until execution-specific authorization is verified? Where would Apple consider the correct enforcement point for such validation: App Intents, the host application, an OS-mediated service, or the actual consequence boundary? Is there an existing Apple framework or security primitive intended to provide this kind of action-specific, pre-effectuation execution authority? Would this model be relevant as Siri and Apple Intelligence gain the ability to perform more cross-app and agentic actions? Could an OS-mediated finality mechanism also provide a common security boundary for first-party and interoperating third-party AI assistants? I am particularly interested in understanding whether this should be considered an App Intents implementation pattern, an operating-system security concern, or a broader architectural mechanism for secure AI interoperability.
0
0
410
Aug ’26
Foundation Model tool calling giving system error in iOS27 beta 5
After updating my iOS and xcode to latest iOS 27 beta5 and xcode 27 beta5 all the system language model session calls with tool calls inclusion throwing Unrecognized system-instruction prefix ID: com.apple.fm_api.tool_calls_override error. The same code was working perfectly in iOS27 beta 4. Even the apple sample project OrigamiCraftingADynamicTutorialForAppleIntelligence failing with the same error when tool calls invoked. Anybody else facing similar issue or any workaround for this issue? sample code: struct GetRecordNotesTool: Tool { let name = "getRecordNotes" let description = "Fetches internal notes and returns Note_Title and Note_Content for up to 10 notes." @Generable struct Arguments { @Guide(description: "The API name of the module, e.g. Companies or Contacts") var module_api_name: String @Guide(description: "The unique record ID to fetch notes for") var record_id: String } func call(arguments: Arguments) async throws -> String { return "Fetched content" } }
1
0
960
Aug ’26
Advice on Referencing Previous Prompts / Responses
When using Private Cloud Compute, I want to be able to submit more than one prompt per LanguageModelSession, ideally using the prompt and response from the first interaction to inform a second interaction. How can I reference this first prompt and response when making a subsequent prompt in a session? I have tried plain language like "current data" and "previous prompt" but it does not seem to understand.
1
0
441
Aug ’26
How are you iterating on Foundation Models prompts before building the app workflow?
While building with Apple's Foundation Models, I kept running into a workflow problem before the app code itself. The hard part was not only calling LanguageModelSession. It was figuring out the shape of the interaction: What should be in the system prompt? What should stay in the user input? What output is actually usable by the app? How much instruction is too much? How do I test the same prompt repeatedly without creating another small Xcode project? I ended up building a small macOS tool for myself, LocalLM Lab, mainly to speed up that loop. The first use case was a Prompt Playground: system prompt, user input, model output, and a repeatable way to compare results before moving the workflow into app code. The current version also experiments with connector-style context, such as system clock, weather, reminders/calendar, contacts, and a scoped filesystem folder. That has made the prompt design problem more interesting, because the question becomes: what context should the model see, and how should the app frame that context so the output is useful? I am curious how other developers are handling this while building with Foundation Models. Are you mostly iterating inside Xcode playgrounds? Are you building small internal test harnesses? Are you separating system prompts and user inputs during testing? How are you evaluating whether the output is reliable enough for the app workflow? For reference, this is the tool I have been using for my own experiments: https://thisbrain.ai/locallm I would be especially interested in any patterns people have found for designing and testing prompts before committing them to app code.
4
0
784
Aug ’26
Can I prevent Siri from "seeing" specific parts of my app?
Let's say my app is a password or other type of secret manager, which means it can display highly sensitive data to the user. If Siri summarize whats on screen, it would then be able to "see" that sensitive data and transmit it off device in a way the user might not expect nor want. Is there a way that I can prevent this by marking specific views in an app as sensitive, or detect if Siri is analyzing the contents of the apps screen? The .privacySensitive() modifier and @Environment(\.isSceneCaptured) environment variable do not seem to apply in this case.
0
0
691
Aug ’26
VNDetectBarcodesRequest fails on every frame: "Could not build inference plan - ANECF error: failed to load ANE model .../mrcdetector.H17.espresso.hwx"
On iPhone 16e running iOS 26, we have now lost barcode detection through two independent APIs. Other device models in the same fleet, on the same app build and the same iOS version, are unaffected. Background: the original failure (AVCaptureMetadataOutput) Our retail app scans EAN-13, Code 128 and ITF barcodes. It originally used AVCaptureMetadataOutput with metadataObjectTypes set accordingly. After the update to iOS 26, this stopped working on iPhone 16e. The behaviour was completely silent: the capture session reported isRunning == true, the camera preview stayed live and correctly exposed, no interruption or runtime-error notifications were posted — but metadataOutput(_:didOutput:from:) simply never fired again, for any barcode. There was no error of any kind to go on. Restarting the app did not help. Only a full device reboot restored detection. Because the metadata path performs detection in the media daemon rather than in our process, we moved detection into the app to work around it. The current failure (Vision) Frames now come from an AVCaptureVideoDataOutput (preset .hd1280x720, .up orientation) and are analyzed in-process: let request = VNDetectBarcodesRequest() request.symbologies = [...] let handler = VNImageRequestHandler(cvPixelBuffer: pixelBuffer, orientation: .up, options: [:]) try handler.perform([request]) On the same devices, handler.perform() now throws for every analyzed frame: Error Domain=com.apple.Vision Code=9 "Could not build inference plan - ANECF error: failed to load ANE model file:///System/Library/Frameworks/Vision.framework/ mrcdetector.H17.espresso.hwx Error=createProgramInstanceForModel:modelToken: modelFilePath:qos:isPreCompiled:enablePowerSaving:skipPreparePhase:statsMask: memoryPoolID:enableLateLatch:modelIdentityStr:owningPid:cacheUrlIdentifier: aotCacheUrlIdentifier:optOutOfModelMemoryUnwiring:error:: Program load failure (0x50004) (DESIGN)" So the second approach fails as well — but loudly, and one layer down. The failing model is a system model shipped inside Vision.framework; we load no Core ML model of our own. Shared characteristics Both failures behave identically in the ways that matter: Same device model (iPhone 16e), starting with iOS 26 Camera pipeline healthy throughout: frames keep arriving continuously (verified by a watchdog on the sample buffer delegate), preview live, no AVCaptureSessionWasInterrupted and no AVCaptureSessionRuntimeError Detection never recovers on its own An app restart does not help; only a device reboot does Over one workday we recorded ~65,000 consecutive Vision failures across four devices, with zero successful detections in between. This makes us suspect both symptoms share a root cause below the two APIs, rather than being two unrelated bugs. What we tried We found thread 761095, where the same error signature (Code=9, "Could not build inference plan - ANECF error", "(DESIGN)") was reported on visionOS 2.0 beta for a different system model, and where the suggested workaround was to restrict the request to CPU/GPU via setComputeDevice(_:for:). We implemented that as a runtime fallback: let devices = try request.supportedComputeStageDevices[.main] ?? [] // pick .gpu, else .cpu request.setComputeDevice(device, for: .main) We have not yet been able to confirm on an affected device whether this actually bypasses the failing path, since we cannot reproduce the state on demand. Questions Is this a known issue on iOS 26 / iPhone 16e? Could the silent AVCaptureMetadataOutput failure and this ANE model load failure share a common cause? Is restricting the compute stage to CPU/GPU a supported workaround for VNDetectBarcodesRequest, or does the barcode detector always require the ANE? Is there any way to recover the ANE state from within the app, so users do not have to reboot the device? Is there anything specific we should capture in a Feedback Assistant report to make this actionable? We can reproduce it in the field but not on demand.
1
0
668
Aug ’26
Custom AppSchema domains
Apple's strict contracts for App schema domains are great if you have something which fits into that domain. There are endless options with AppSchema domains outside that of what apple have created. Does anyone know if apple will open the door to custom AppSchema domains? This would be a "game-changer". Is there any insight on what the future holds?
1
1
1.3k
Aug ’26
Foundation Models are broken in iOS 27 Beta
Hi guys, I'm testing the Foundation Models Framework with the on-device model in iOS 27 (beta 4) and macOS 27 (beta 4) and is completely failing to respond. There are many errors. For starters, the model doesn't respond to prompts directly, you need to specify instructions, otherwise it refuses to provide an answer. It is always looking for tools, even when no tool has been provided, and returns an error saying that it couldn't find the tool. Then, when it produces a response, it shows all the thinking process first, which completely ruins the response. Most of the time, the response begins with all the JSON code. And when I try to have a long conversation, it just says "I cannot write content or generate text." I wonder if someone is experiencing the same issues or maybe the way to implement this model changed and I'm missing something? Here is a screenshot of one of my interactions when I asked the model to describe a unicorn. It tried to access a tool that doesn't exist. (the app just prints the value of the content property) Here is the code. It is performing a simple request. struct ContentView: View { @State private var response = "" var body: some View { VStack { Button("Send") { let prompt = "Write a paragraph describing a unicorn" let session = LanguageModelSession { "Respond to the user's request. Never acknowledge the request, add preamble, or comment on what you are about to write." } if !session.isResponding { Task { do { let answer = try await session.respond(to: prompt) response = answer.content } catch { response = "Error accessing the model: \(error)" } } } } .buttonStyle(.borderedProminent) Text(response) .font(Font.system(size: 18)) .padding() Spacer() } .padding() } }
5
0
1.7k
Aug ’26
Use of SpotlightSearchTool() returns "Model Catalog error: Error Domain=com.apple.UnifiedAssetFramework Code=5000" , although model is available
On macOS Golden Gate Developer Beta 4 the following code: import CoreSpotlight import FoundationModels let tool = SpotlightSearchTool() let session = LanguageModelSession(tools: [tool]) let response = try await session.respond(to: "What hikes have I gone on?") , returns the following error: Model Catalog error: Error Domain=com.apple.UnifiedAssetFramework Code=5000 "There are no underlying assets (neither atomic instance nor asset roots) for consistency token for asset set com.apple.modelcatalog" UserInfo={NSLocalizedFailureReason=There are no underlying assets (neither atomic instance nor asset roots) for consistency token for asset set com.apple.modelcatalog} , although the model is available in general and can return responses without using the tool. The code: print(SystemLanguageModel.default.availability) returns 'available'. What am I doing wrong?
9
0
1.4k
Aug ’26
Is there any way to re-enable App Shortcuts Siri opt-in on watchOS after it has been declined?
On watchOS, once the "Turn on [App] shortcuts with Siri?" prompt has been declined for an app, is there a user-facing way to turn it back on? On iOS I found the per-app Siri toggle, and re-enabling it restored voice invocation immediately. I cannot find any equivalent on watchOS, in the watch Shortcuts app, in watch Settings, or in the paired iPhone's Watch app. If one exists I would appreciate a pointer. If it does not, I would like to know whether that is expected. What I am seeing I have an iOS app with a companion watchOS app. Each declares its own AppShortcutsProvider and App Intent. Modern App Intents only, no SiriKit, no Intents extension. On one Apple Watch, saying the trigger phrase transcribes correctly on screen, then falls through to a generic Maps business search. The intent is never invoked. On that same watch the App Shortcut appears in the watch Shortcuts app and runs correctly when tapped, and Siri is otherwise healthy. The same phrase works on the paired iPhone. Why I believe the app is fine On a second iPhone and Apple Watch on a different Apple Account, the phrase spoken to the watch returned my intent's own dialog from perform(). Two control apps also behave correctly, including one with an iOS host plus embedded companion watch app matching my production structure, which prompted and worked on both devices. Reading Metadata.appintents/extract.actionsdata from the built watch app shows the provider registered and all phrase templates well formed. The state survives deleting the app. I deleted it from both devices, power cycled both, and reinstalled. The iPhone worked immediately without re-prompting. The watch continued to fail without prompting. Both retained their prior opt-in state. The app's bundle identifiers changed during a rename several months ago. I believe the prompt appeared on the watch then and was dismissed. Environment Apple Watch Ultra, watchOS 26.6. iPhone on iOS 18.6. Xcode 26.1.1. Deployment targets iOS 18.6 / watchOS 11.6. Companion watch app, WKRunsIndependentlyOfCompanionApp = NO. Already ruled out on device, each with a clean install and watch restart: CFBundleSpokenName, one word and spaced Watch PRODUCT_NAME / CFBundleName matched to the spoken app name Removing updateAppShortcutParameters() from the watch App.init() Adding the com.apple.developer.siri entitlement to the watch target Phrase collision between phone and watch providers, a watch-only phrase also failed Siri domain collision on the trigger words, a neutral phrase also failed Intent identifier collisions between the phone and watch bundles A brand new, uniquely named intent with canonical syntax and no gating was also ignored Questions Is there a user-facing setting on watchOS, or in the paired iPhone's Watch app, to re-enable App Shortcuts for a specific app after the prompt has been declined? If not, is there any supported recovery short of erasing the watch and setting it up as new? Restoring from backup appears to bring the state back. Is there any API for an app to detect that its App Shortcuts opt-in is off, so it can show guidance instead of a silently dead feature? INPreferences.siriAuthorizationStatus() is not a valid proxy, it reports authorized on the affected device. Happy to provide a sysdiagnose with the App Intents logging profile, or either control project.
2
0
795
Aug ’26
Generation Error
So I'm having an issue with the FoundationModels framework but idk if this is just my feeling or not, the issue comes up after I updated my Mac into 26.6 the code was very simple actually: #Playground { let model = SystemLanguageModel.default let session = LanguageModelSession(model: model) print(model.availability) var query = "How to hide button" Task { do { let response = try await session.respond(to: query) print(response.content) } catch { print("\(error)") } } } the code works before I updated the version, but then after I updated the version it says: Error Domain=FoundationModels.LanguageModelSession.GenerationError Code=-1 "The operation couldn’t be completed. (FoundationModels.LanguageModelError error -1.)" UserInfo={NSMultipleUnderlyingErrorsKey=( "Error Domain=FoundationModels.LanguageModelError Code=-1 "(null)" UserInfo={NSMultipleUnderlyingErrorsKey=(\n "Error Domain=ModelManagerServices.ModelManagerError Code=1026 \"(null)\" UserInfo={NSMultipleUnderlyingErrorsKey=(\n)}"\n)}" ), NSLocalizedDescription=The operation couldn’t be completed. (FoundationModels.LanguageModelError error -1.)} this is runned in Xcode 26.6, additional information I have also coder 27 beta 4 installed in my Mac, is this problem occurring because the Xcode 26.6 and Xcode 27 beta 4?? can u guys help me
2
0
900
Aug ’26
Fused Metal Kernels for Linear Recurrences in MLX
I’ve been developing mlx-recurrence, a plug-in framework of fused Metal GPU kernels for linear recurrences on Apple silicon—roughly analogous to flash linear attention for MLX. Sequential recurrences are difficult for MLX to fuse automatically. Architectures such as state-space models, gated linear attention, and diagonal RNNs ordinarily require a loop across the sequence length. When that loop is implemented in Python, a sequence of length L can require L separate Python-to-Metal dispatches. These kernels instead execute the entire recurrence in a single Metal dispatch. The training path uses segment checkpointing with recomputation during the backward pass. In validated M3 Max tests, the checkpoint-and-recompute kernels reduced peak recurrent-state memory by approximately 12–18× at the kernel level and lowered total training peak memory from 23.88 GB to 10.34 GB. At the same batch size, end-to-end training throughput improved by roughly 1.4×, while individual fused forward-and-backward kernels ran approximately 1.5–1.9× faster than the original full-state implementations. Results will vary with recurrence type, sequence length, state dimensions, batch size, datatype, model architecture, and hardware. Current kernels: ssd_scan Mamba-2-style, head-wise SSD selective scan. Intended for Mamba-2 and other SSM hybrid architectures. State shape: [B, H, Dh, N] gla_scan Gated Linear Attention with a scalar forget gate and outer-product write. Intended for GLA and linear-attention hybrid architectures. State shape: [B, H, Dh, Dh] rglru_scan RG-LRU diagonal recurrence. Intended for Griffin and RecurrentGemma-style architectures. State shape: [B, D] rotlru_scan Rotational LRU using a complex-diagonal recurrence, a magnitude gate, and a per-step rotation of two-dimensional channel pairs. Intended for complex-LRU and S4-style oscillatory memory architectures. State shape: [B, D], represented as interleaved channel pairs. Each kernel is implemented as a self-contained plug-in on a shared chassis located at: mlx_recurrence._chassis The chassis provides: Segment checkpoint-and-recompute infrastructure Shape and argument validation VJP integration Forward and gradient parity-test helpers Common recurrence plug-in handling Adding another recurrence therefore requires implementing its Metal forward and backward source pair and connecting its VJP. The checkpointing, validation, and testing infrastructure does not need to be rebuilt for each operator. The original version 0.1 kernels remain available under: mlx_recurrence.legacy They are also re-exported at the package’s top level for backward compatibility. I’m interested in feedback from developers working with MLX or custom Metal compute kernels, particularly around: Preferred APIs for packaging reusable MLX Metal extensions Threadgroup and memory-layout strategies across Apple GPU generations Numerical stability expectations for long recurrent sequences Benchmarking fused scans against MLX-native implementations Additional recurrent operators that would be valuable to support I would also be interested to know whether others are developing similar fused recurrence primitives for MLX and whether a shared interface for these operations would be useful. My setup is a M3 MAX Macbook Pro with 36GB Ram and I am running on macOS 26.4.1 (25E253). https://github.com/D-CSIL/mlx-recurrence
0
0
307
Aug ’26
Xcode treats a `.llmasset` bundle as individual `.aimodel` files to compile, instead of copying it as-is
I have a Core AI model export — a bundle folder (.llmasset, containing multiple .aimodel subfolders plus metadata/tokenizer resources) — added to my app target as a folder reference. Rather than treating the bundle as one opaque resource and copying it into the app bundle as-is (the way .xcassets, .bundle, or any other folder reference behaves), Xcode reaches into it, finds the individual .aimodel subfolders, and adds each one to Compile Sources. When it compiles them there, it's for my build machine's specific chip only — I can't find any setting (Build Settings, scheme, target picker) to compile for multiple architectures/platforms, the way a universal binary would work. Question: Is there a way to make Xcode treat a .llmasset bundle as an atomic resource — copied wholesale, not decomposed into individual .aimodel compile targets? Or is reaching into the bundle and AOT-compiling its components for the active build architecture the intended behavior here, and if so, what's the recommended way to make sure the result works across the actual range of devices the app ships to?
1
0
470
Aug ’26
AppIntent ignores registered dependencies when awaited
App intent has a perform method that is async and can throw an error, but I can't find a way to actually await the result and catch the error if needed. If I convert this working but non-waiting, non-catching code: Button("Go", intent: MyIntent()) to this (so I can control awaiting and error handling): Button("Go") { Task { do { try await MyIntent().perform() // 👈 } catch { print(error) } } } It crashes: AppDependency with key "foo" of type Bar.Type was not initialized prior to access. Dependency values can only be accessed inside of the intent perform flow and within types conforming to _SupportsAppDependencies unless the value of the dependency is manually set prior to access. Although it is invalid since the first version is working like a charm and dependencies are registered in the @main App init method and it is in the perform flow. So how can we await the result of the AppIntent and handle the errors if needed in the app? Should I re-invent the Dependency mechanism?
Replies
1
Boosts
0
Views
1.1k
Activity
Aug ’26
MLXVLM factory fallback doesn't catch a vision_config-present-but-no-tower-weights checkpoint (Qwen3.5-9B)
I ran into a gap in the MLXVLM → MLXLLM factory fallback and wanted to check whether this is expected behavior or something worth filing. Context: I maintain a Swift-native macOS agent built on MLX Swift, with a hardware-adaptive catalog of local models. I tried adding mlx-community/Qwen3.5-9B-4bit as a plain text model. Its config.json ships a full vision_config block — it looks like a multimodal config — but the actual checkpoint has no vision tower weights at all. It's genuinely text-only; the metadata just doesn't reflect that. ModelFactoryRegistry tries MLXVLM before falling back to MLXLLM. I expected that to fail cleanly and fall through, since there's no vision encoder to load. It doesn't. MLXVLM's Qwen35Configuration declares visionConfiguration as a non-optional property mapped to vision_config. Since the key is present in the raw config, decoding succeeds — there's nothing to catch at that stage. So MLXVLM proceeds to actually load the model as a VLM, and it crashes later, inside WiredMemoryUtils.tune(), because getRopeIndex() expects real vision-encoder dimensions to compute against, and there are none. Not a clean decode-time error I could catch — a runtime crash several layers past config parsing, in the RoPE indexing math. The workaround I ended up with is config-level surgery before the model reaches the factory at all: strip vision_config (and image_token_id / video_token_id) from the JSON first. With the key genuinely absent, Qwen35Configuration's decode throws DecodingError.keyNotFound as the fallback logic presumably intends, MLXVLM fails cleanly at that stage, and ModelFactoryRegistry falls through to MLXLLM correctly. That works, but it feels like a workaround rather than the intended path — the factory fallback seems to assume any failure will happen at decode time, and a config key that's present but semantically empty (no real vision weights behind it) skips that safety net entirely. Questions: Is there a more canonical/official way to detect whether a checkpoint actually has real vision tower weights, short of opening model.safetensors.index.json and counting vision_tower.* tensor names myself? (I've done exactly that for a different, genuinely multimodal Qwen VL checkpoint — 333 real vision tensors there vs. zero here — but it feels like something the loading path should be able to tell me.) Is the vision_config-present-but-empty pattern something the mlx-community conversion pipeline is aware of, or is this specific to how Qwen3.5-9B-4bit happened to get converted? Happy to share more logs/repro details if useful.
Replies
0
Boosts
0
Views
409
Activity
Aug ’26
How can a local AI agent use MLX/Metal unattended on macOS while remaining confined to an authorized workspace?
How can a local AI agent use MLX/Metal unattended while remaining confined to an authorized workspace? I am developing an AI-driven local media-processing workflow on an Apple-silicon Mac and am trying to understand the correct architecture for allowing it to run unattended without giving the AI agent unrestricted access to my primary personal computer. I am not a software engineer, so I may be missing an established macOS mechanism or using the wrong terminology. I would appreciate guidance from people familiar with MLX, Metal, sandboxing, and macOS security. What I am building I use OpenAI Codex as the local execution/software-development agent. The working system currently: ingests and verifies original video and still media while preserving immutable originals; performs visual semantic analysis and divides video into meaningful time-coded segments; separately analyzes spoken language rather than assuming audio and video are semantically equivalent; uses MLX Whisper locally on Apple silicon for time-coded speech transcription; stores visual and language semantics in a relational SQLite media catalog. These five stages are working. My current test corpus contains 148 original media files, 126 visual semantic segments, and 765 speech segments. The next stages are AI editorial construction from the semantic database and generation of instructions/scripts for a DaVinci Resolve rough cut. The security architecture I want Codex to operate autonomously within a deliberately bounded development environment. I do not want to solve this simply by granting an autonomous agent Full Disk Access to my primary personal Mac. The concern is ordinary fault containment. Codex generates and executes scripts, invokes applications and command-line tools, and manipulates files. A mistaken path or defective generated script should not have unrestricted consequences for the rest of my computer. I therefore separated AI execution from ordinary personal files. Codex is configured for Workspace Write access with explicitly authorized project roots. Canonical media resides on a separately authorized external SSD, and temporary AI working artifacts are kept separately. Ordinary Python and FFmpeg operations now run autonomously within these authorized areas. The problem The difficulty appears when the workflow invokes capabilities that cannot operate inside the ordinary Codex sandbox. The clearest example is MLX Whisper. I am using: MLX Whisper 0.4.3 mlx-community/whisper-small-mlx Apple silicon local transcription MLX Whisper works successfully and its transcription quality is sufficient for my semantic-retrieval application. However, MLX could not access Apple Metal/GPU execution from inside the ordinary Codex sandbox. Codex therefore requested permission to execute the transcription operation outside the sandbox. Once approved, MLX/Metal worked and the entire corpus was successfully transcribed. The processing therefore works, but the workflow cannot run genuinely unattended. A future operation should be able to run: new media → integrity verification → visual semantic analysis → MLX Whisper transcription → language semantic analysis → SQLite update → QA But if execution stops midway waiting for a human to click Allow, the pipeline is not operationally autonomous. What I have already tried I initially encountered permission problems even with ordinary file operations. I therefore: separated Codex work from ordinary personal documents; created dedicated project/work areas; explicitly authorized the required working roots; configured Workspace Write; separately authorized the external media repository; tested shell/Python and FFmpeg operations within those boundaries. Those changes worked. Routine Python and FFmpeg operations now run without approval prompts. The remaining issue occurs with MLX/Metal and some other application/runtime operations that require sandbox escalation. My question Is there a supported architecture for allowing a local AI agent to invoke MLX/Metal and other deliberately authorized development tools unattended, while still confining the agent to defined project/workspace boundaries rather than granting unrestricted access to the entire Mac? For example, should I be investigating: App Sandbox entitlements; a signed helper tool or XPC service; security-scoped resources; a dedicated executable with appropriate entitlements; a different method of launching MLX/Metal; or another macOS mechanism? In particular, can Metal/GPU access coexist with persistent bounded filesystem access without requiring interactive approval each time the AI invokes it? I am also unsure which security layer is actually responsible here: the Codex sandbox, macOS App Sandbox, TCC, executable/code-signing rules, Metal restrictions, or some interaction among them. If this kind of bounded unattended execution is intentionally not supported, that would also be useful to know. My alternative would be a dedicated Apple-silicon Mac containing only the AI-development environment and replaceable project data, where broader permissions would have a much smaller failure domain. I can provide the Codex configuration, exact successful and failing commands, directory/root configuration, macOS/hardware information, and sandbox diagnostics. I would particularly appreciate guidance on which security layer is causing the MLX/Metal escalation and what the supported architecture would be for this use case. Thank you.
Replies
0
Boosts
0
Views
899
Activity
Aug ’26
Enhancement: pre-action policy hook before multi-step App Intent / Siri AI perform()
As App Intents power multi-step workflows via Siri, Shortcuts, Spotlight, and Apple Intelligence, I would like to request a platform pre-action policy surface: Before intent perform(): input: intent_id, parameters, caller_agent_id, session_id, risk_hints output: allow | warn | deny | require_confirmation side: local audit_receipt Why this is needed Sandbox and TCC solve app isolation and one-time permission grants. Multi-step agentic loops introduce a different risk shape: Destructive tool selection from noisy natural language2. Uncapped retry / network loops3. Unauthorized data movement across app boundaries4. Financial / identity / Wallet actions that need dual control even when the app is already authorized Prompt-only or documentation-only guidance is not enforceable mid-loop. Apple-aligned properties On-device evaluation by default (privacy)- Complements TCC / sandbox â does not replace them- User-visible WARN / DENY with recoverable explanation- Optional Instruments-style â agent action timelineâ with gate verdicts Risk classes (map to entitlements mental model) | Class | Example | Default posture || --- | --- | --- || Read local | calendar, on-screen text | Allow / low friction || Cross-app write | paste into finance app | Warn or confirm || Destructive | delete, wipe, revoke | Deny or hard confirm || Network exfil | send sensitive context off-device | Deny until confirm || Financial / Wallet | transfer, mint pass | Always confirm | Related discussion There is already an excellent thread on pre-effectuation / execution finality for high-consequence App Intents. This request is complementary: a first-class, developer-visible pre-action hook (ALLOW / WARN / DENY / CONFIRM) at the intentâ action boundary, plus local audit receipts. Shipping reference (independent OSS control plane) I ship ThumbGate (npm thumbgate) â a local-first pre-action firewall used today for AI coding agents: PreToolUse / MCP interception, ALLOW/WARN/DENY, thumbs feedback â prevention rules. Public: https://thumbgate.ai · https://github.com/IgorGanapolsky/ThumbGate I will also file this as a Feedback Assistant Suggestion. Looking for: Confirmation this belongs under App Intents / Apple Intelligence2. Any existing API I missed for third-party or system-level pre-perform gates3. The right internal owner / Feedback area if this should go through a different channel Happy to share a one-pager with the risk taxonomy and proposed hook shape.
Replies
0
Boosts
0
Views
607
Activity
Aug ’26
iPadOS 27 Beta — Siri AI overlay causes no Scene lifecycle callbacks, starves BT data processing threads
Environment: iPadOS 27 Beta (Developer Beta) iPad with Bluetooth Classic (iAP2/ExternalAccessory) + BLE active session App uses UIKit, WKWebView, scene-based lifecycle Problem: When the user invokes the new Siri AI by long-pressing the power button while our app is in the foreground with an active Bluetooth Classic session, we observe: No scene lifecycle callbacks fire — no sceneWillResignActive, no sceneDidEnterBackground, nothing. We confirmed by logging every UISceneDelegate method. Main thread / data processing threads are starved for ~2 seconds, causing a backlog of incoming Bluetooth data. Our real-time data processing latency jumps from ~105ms to over 2,300ms within 2 seconds of Siri activation. CADisplayLink / requestAnimationFrame callbacks show a ~935ms gap coinciding with the Siri overlay appearance, then irregular intervals afterward. The Bluetooth Classic transport (ExternalAccessory/iAP2) remains physically connected throughout — the issue is purely host-side processing starvation. What we've ruled out: BLE link degradation: firmware-side diagnostics confirm 100% data delivery, 0 lost packets during the incident Memory pressure from our app: our process memory stays flat; system-available memory drops ~14 units externally Questions: Is the absence of sceneWillResignActive when Siri AI activates on iPadOS 27 intended behavior, or a beta bug? The new UIApplication.systemPrefersReducedResourceUsage property (iPadOS 27 beta) — is this intended to signal system overlays like Siri consuming resources? Does the corresponding systemPrefersReducedResourceUsageDidChangeNotification fire when Siri activates? Are there recommended patterns for apps with real-time Bluetooth data processing to maintain thread priority during system overlays? We currently use default QoS for our data processing dispatch queues. The processing starvation causes the waveform display to degrade (appears as a connectivity issue to the clinician) even though the wireless link is healthy. We need either: A notification that a system overlay is active, so we can adjust our UI accordingly Guidance on maintaining processing priority during Siri AI activation Any community insight on workarounds would be highly appreciated. Thanks.
Replies
2
Boosts
0
Views
1.5k
Activity
Aug ’26
The new beta Siri disappeared
I've already used the new Siri for a few weeks but now it disappeared. The Siri app still exists but cannot use. Currently the Siri is the original Apple Intelligence. I went to Settings, Screen Time, Content & Privacy Restrictions, Siri, and the Siri AI (Beta) is grey cannot choose. Anyone knows how to fix it?
Replies
0
Boosts
1
Views
370
Activity
Aug ’26
Pre-Effectuation Execution Finality for Siri and App Intents
I would like to discuss a possible security architecture for Siri, Apple Intelligence, and App Intents where permission to invoke an app action is separated from permission for that specific action to become externally effective. For low-risk actions, existing authorization may be sufficient. However, for higher-consequence actions—such as payments, file export, message transmission, account changes, device control, or other irreversible operations—there may be value in introducing an additional execution-finality boundary. Problem Space An AI assistant may be authorized to invoke an App Intent, but that does not necessarily mean every resulting action should immediately become effective. For example: Siri may be allowed to invoke a payment-related intent, but not every amount or recipient should necessarily be executable. An app may expose a file-sharing intent, but a particular file or destination may fall outside the permitted scope. An AI-generated message may be validly created, but its final transmission may require additional execution-specific validation. Context, permission, destination, revocation state, or security state may change between intent generation and actual execution. The proposed distinction is: Permission to invoke an App Intent versus Permission for the specific resulting act to become externally effective Proposed Architecture A high-consequence action could first become a Candidate Act and remain in a Non-Effective State. Conceptually: Siri / Apple Intelligence ↓ App Intent ↓ Candidate Act ↓ Non-Effective State ↓ Protected Validation ↓ Scoped Execution Authority ↓ Finality Sink Verification ↓ External Effect If validation fails, expires, is revoked, is replayed, or becomes ambiguous: Default Denial → No External Effect Short Definitions Candidate Act The specific operation proposed by Siri, Apple Intelligence, or an app before it is allowed to create an external consequence. Non-Effective State A state in which the operation may be prepared, inspected, or evaluated but cannot yet produce its intended external effect. Protected Validation A validation step checking execution-relevant conditions such as app identity, user authorization, purpose, destination, scope, freshness, limits, revocation state, or device security state. Scoped Execution Authority Authority limited to the specific validated action rather than a broadly reusable permission. Finality Sink The consequence boundary where the action becomes externally effective—for example, a network transmission, file release, payment commit, database change, or physical-device action. Why This Could Be Useful This architecture could provide: separation of AI decision-making from final execution authority; fail-closed behavior for invalid or uncertain actions; resistance to replay or stale authorization; action-specific rather than broadly reusable authority; validation closer to the actual consequence boundary; stronger control for increasingly autonomous AI workflows. This would be intended as an optional mechanism for higher-consequence actions, not as a replacement for App Intents, existing user authorization, entitlements, sandboxing, or other Apple security mechanisms. Possible Interoperability Relevance This question may also become relevant as operating-system interoperability requirements evolve, including in the European Union under the Digital Markets Act. As third-party AI assistants and services gain deeper interoperability with operating-system features, there may be a need to distinguish between allowing an interoperating service to request an action and allowing that specific action to cross the final consequence boundary. A device-side execution-finality mechanism could potentially provide a technical middle layer: third-party AI services could request interoperable actions, while the operating system retains a neutral protected mechanism for validating the specific action immediately before it becomes externally effective. This may help explore how broader interoperability and strong device-side security could coexist without requiring unrestricted execution authority for either first-party or third-party AI assistants. I would be interested in whether Apple considers this type of consequence-boundary enforcement compatible with existing or future App Intents and interoperability architectures. Questions for Apple Engineers and Developers Does App Intents currently provide a supported mechanism for maintaining an action in a non-effective state until execution-specific authorization is verified? Where would Apple consider the correct enforcement point for such validation: App Intents, the host application, an OS-mediated service, or the actual consequence boundary? Is there an existing Apple framework or security primitive intended to provide this kind of action-specific, pre-effectuation execution authority? Would this model be relevant as Siri and Apple Intelligence gain the ability to perform more cross-app and agentic actions? Could an OS-mediated finality mechanism also provide a common security boundary for first-party and interoperating third-party AI assistants? I am particularly interested in understanding whether this should be considered an App Intents implementation pattern, an operating-system security concern, or a broader architectural mechanism for secure AI interoperability.
Replies
0
Boosts
0
Views
410
Activity
Aug ’26
Foundation Model tool calling giving system error in iOS27 beta 5
After updating my iOS and xcode to latest iOS 27 beta5 and xcode 27 beta5 all the system language model session calls with tool calls inclusion throwing Unrecognized system-instruction prefix ID: com.apple.fm_api.tool_calls_override error. The same code was working perfectly in iOS27 beta 4. Even the apple sample project OrigamiCraftingADynamicTutorialForAppleIntelligence failing with the same error when tool calls invoked. Anybody else facing similar issue or any workaround for this issue? sample code: struct GetRecordNotesTool: Tool { let name = "getRecordNotes" let description = "Fetches internal notes and returns Note_Title and Note_Content for up to 10 notes." @Generable struct Arguments { @Guide(description: "The API name of the module, e.g. Companies or Contacts") var module_api_name: String @Guide(description: "The unique record ID to fetch notes for") var record_id: String } func call(arguments: Arguments) async throws -> String { return "Fetched content" } }
Replies
1
Boosts
0
Views
960
Activity
Aug ’26
Advice on Referencing Previous Prompts / Responses
When using Private Cloud Compute, I want to be able to submit more than one prompt per LanguageModelSession, ideally using the prompt and response from the first interaction to inform a second interaction. How can I reference this first prompt and response when making a subsequent prompt in a session? I have tried plain language like "current data" and "previous prompt" but it does not seem to understand.
Replies
1
Boosts
0
Views
441
Activity
Aug ’26
How are you iterating on Foundation Models prompts before building the app workflow?
While building with Apple's Foundation Models, I kept running into a workflow problem before the app code itself. The hard part was not only calling LanguageModelSession. It was figuring out the shape of the interaction: What should be in the system prompt? What should stay in the user input? What output is actually usable by the app? How much instruction is too much? How do I test the same prompt repeatedly without creating another small Xcode project? I ended up building a small macOS tool for myself, LocalLM Lab, mainly to speed up that loop. The first use case was a Prompt Playground: system prompt, user input, model output, and a repeatable way to compare results before moving the workflow into app code. The current version also experiments with connector-style context, such as system clock, weather, reminders/calendar, contacts, and a scoped filesystem folder. That has made the prompt design problem more interesting, because the question becomes: what context should the model see, and how should the app frame that context so the output is useful? I am curious how other developers are handling this while building with Foundation Models. Are you mostly iterating inside Xcode playgrounds? Are you building small internal test harnesses? Are you separating system prompts and user inputs during testing? How are you evaluating whether the output is reliable enough for the app workflow? For reference, this is the tool I have been using for my own experiments: https://thisbrain.ai/locallm I would be especially interested in any patterns people have found for designing and testing prompts before committing them to app code.
Replies
4
Boosts
0
Views
784
Activity
Aug ’26
Can I prevent Siri from "seeing" specific parts of my app?
Let's say my app is a password or other type of secret manager, which means it can display highly sensitive data to the user. If Siri summarize whats on screen, it would then be able to "see" that sensitive data and transmit it off device in a way the user might not expect nor want. Is there a way that I can prevent this by marking specific views in an app as sensitive, or detect if Siri is analyzing the contents of the apps screen? The .privacySensitive() modifier and @Environment(\.isSceneCaptured) environment variable do not seem to apply in this case.
Replies
0
Boosts
0
Views
691
Activity
Aug ’26
VNDetectBarcodesRequest fails on every frame: "Could not build inference plan - ANECF error: failed to load ANE model .../mrcdetector.H17.espresso.hwx"
On iPhone 16e running iOS 26, we have now lost barcode detection through two independent APIs. Other device models in the same fleet, on the same app build and the same iOS version, are unaffected. Background: the original failure (AVCaptureMetadataOutput) Our retail app scans EAN-13, Code 128 and ITF barcodes. It originally used AVCaptureMetadataOutput with metadataObjectTypes set accordingly. After the update to iOS 26, this stopped working on iPhone 16e. The behaviour was completely silent: the capture session reported isRunning == true, the camera preview stayed live and correctly exposed, no interruption or runtime-error notifications were posted — but metadataOutput(_:didOutput:from:) simply never fired again, for any barcode. There was no error of any kind to go on. Restarting the app did not help. Only a full device reboot restored detection. Because the metadata path performs detection in the media daemon rather than in our process, we moved detection into the app to work around it. The current failure (Vision) Frames now come from an AVCaptureVideoDataOutput (preset .hd1280x720, .up orientation) and are analyzed in-process: let request = VNDetectBarcodesRequest() request.symbologies = [...] let handler = VNImageRequestHandler(cvPixelBuffer: pixelBuffer, orientation: .up, options: [:]) try handler.perform([request]) On the same devices, handler.perform() now throws for every analyzed frame: Error Domain=com.apple.Vision Code=9 "Could not build inference plan - ANECF error: failed to load ANE model file:///System/Library/Frameworks/Vision.framework/ mrcdetector.H17.espresso.hwx Error=createProgramInstanceForModel:modelToken: modelFilePath:qos:isPreCompiled:enablePowerSaving:skipPreparePhase:statsMask: memoryPoolID:enableLateLatch:modelIdentityStr:owningPid:cacheUrlIdentifier: aotCacheUrlIdentifier:optOutOfModelMemoryUnwiring:error:: Program load failure (0x50004) (DESIGN)" So the second approach fails as well — but loudly, and one layer down. The failing model is a system model shipped inside Vision.framework; we load no Core ML model of our own. Shared characteristics Both failures behave identically in the ways that matter: Same device model (iPhone 16e), starting with iOS 26 Camera pipeline healthy throughout: frames keep arriving continuously (verified by a watchdog on the sample buffer delegate), preview live, no AVCaptureSessionWasInterrupted and no AVCaptureSessionRuntimeError Detection never recovers on its own An app restart does not help; only a device reboot does Over one workday we recorded ~65,000 consecutive Vision failures across four devices, with zero successful detections in between. This makes us suspect both symptoms share a root cause below the two APIs, rather than being two unrelated bugs. What we tried We found thread 761095, where the same error signature (Code=9, "Could not build inference plan - ANECF error", "(DESIGN)") was reported on visionOS 2.0 beta for a different system model, and where the suggested workaround was to restrict the request to CPU/GPU via setComputeDevice(_:for:). We implemented that as a runtime fallback: let devices = try request.supportedComputeStageDevices[.main] ?? [] // pick .gpu, else .cpu request.setComputeDevice(device, for: .main) We have not yet been able to confirm on an affected device whether this actually bypasses the failing path, since we cannot reproduce the state on demand. Questions Is this a known issue on iOS 26 / iPhone 16e? Could the silent AVCaptureMetadataOutput failure and this ANE model load failure share a common cause? Is restricting the compute stage to CPU/GPU a supported workaround for VNDetectBarcodesRequest, or does the barcode detector always require the ANE? Is there any way to recover the ANE state from within the app, so users do not have to reboot the device? Is there anything specific we should capture in a Feedback Assistant report to make this actionable? We can reproduce it in the field but not on demand.
Replies
1
Boosts
0
Views
668
Activity
Aug ’26
Custom AppSchema domains
Apple's strict contracts for App schema domains are great if you have something which fits into that domain. There are endless options with AppSchema domains outside that of what apple have created. Does anyone know if apple will open the door to custom AppSchema domains? This would be a "game-changer". Is there any insight on what the future holds?
Replies
1
Boosts
1
Views
1.3k
Activity
Aug ’26
Foundation Models are broken in iOS 27 Beta
Hi guys, I'm testing the Foundation Models Framework with the on-device model in iOS 27 (beta 4) and macOS 27 (beta 4) and is completely failing to respond. There are many errors. For starters, the model doesn't respond to prompts directly, you need to specify instructions, otherwise it refuses to provide an answer. It is always looking for tools, even when no tool has been provided, and returns an error saying that it couldn't find the tool. Then, when it produces a response, it shows all the thinking process first, which completely ruins the response. Most of the time, the response begins with all the JSON code. And when I try to have a long conversation, it just says "I cannot write content or generate text." I wonder if someone is experiencing the same issues or maybe the way to implement this model changed and I'm missing something? Here is a screenshot of one of my interactions when I asked the model to describe a unicorn. It tried to access a tool that doesn't exist. (the app just prints the value of the content property) Here is the code. It is performing a simple request. struct ContentView: View { @State private var response = "" var body: some View { VStack { Button("Send") { let prompt = "Write a paragraph describing a unicorn" let session = LanguageModelSession { "Respond to the user's request. Never acknowledge the request, add preamble, or comment on what you are about to write." } if !session.isResponding { Task { do { let answer = try await session.respond(to: prompt) response = answer.content } catch { response = "Error accessing the model: \(error)" } } } } .buttonStyle(.borderedProminent) Text(response) .font(Font.system(size: 18)) .padding() Spacer() } .padding() } }
Replies
5
Boosts
0
Views
1.7k
Activity
Aug ’26
Use of SpotlightSearchTool() returns "Model Catalog error: Error Domain=com.apple.UnifiedAssetFramework Code=5000" , although model is available
On macOS Golden Gate Developer Beta 4 the following code: import CoreSpotlight import FoundationModels let tool = SpotlightSearchTool() let session = LanguageModelSession(tools: [tool]) let response = try await session.respond(to: "What hikes have I gone on?") , returns the following error: Model Catalog error: Error Domain=com.apple.UnifiedAssetFramework Code=5000 "There are no underlying assets (neither atomic instance nor asset roots) for consistency token for asset set com.apple.modelcatalog" UserInfo={NSLocalizedFailureReason=There are no underlying assets (neither atomic instance nor asset roots) for consistency token for asset set com.apple.modelcatalog} , although the model is available in general and can return responses without using the tool. The code: print(SystemLanguageModel.default.availability) returns 'available'. What am I doing wrong?
Replies
9
Boosts
0
Views
1.4k
Activity
Aug ’26
Is there any way to re-enable App Shortcuts Siri opt-in on watchOS after it has been declined?
On watchOS, once the "Turn on [App] shortcuts with Siri?" prompt has been declined for an app, is there a user-facing way to turn it back on? On iOS I found the per-app Siri toggle, and re-enabling it restored voice invocation immediately. I cannot find any equivalent on watchOS, in the watch Shortcuts app, in watch Settings, or in the paired iPhone's Watch app. If one exists I would appreciate a pointer. If it does not, I would like to know whether that is expected. What I am seeing I have an iOS app with a companion watchOS app. Each declares its own AppShortcutsProvider and App Intent. Modern App Intents only, no SiriKit, no Intents extension. On one Apple Watch, saying the trigger phrase transcribes correctly on screen, then falls through to a generic Maps business search. The intent is never invoked. On that same watch the App Shortcut appears in the watch Shortcuts app and runs correctly when tapped, and Siri is otherwise healthy. The same phrase works on the paired iPhone. Why I believe the app is fine On a second iPhone and Apple Watch on a different Apple Account, the phrase spoken to the watch returned my intent's own dialog from perform(). Two control apps also behave correctly, including one with an iOS host plus embedded companion watch app matching my production structure, which prompted and worked on both devices. Reading Metadata.appintents/extract.actionsdata from the built watch app shows the provider registered and all phrase templates well formed. The state survives deleting the app. I deleted it from both devices, power cycled both, and reinstalled. The iPhone worked immediately without re-prompting. The watch continued to fail without prompting. Both retained their prior opt-in state. The app's bundle identifiers changed during a rename several months ago. I believe the prompt appeared on the watch then and was dismissed. Environment Apple Watch Ultra, watchOS 26.6. iPhone on iOS 18.6. Xcode 26.1.1. Deployment targets iOS 18.6 / watchOS 11.6. Companion watch app, WKRunsIndependentlyOfCompanionApp = NO. Already ruled out on device, each with a clean install and watch restart: CFBundleSpokenName, one word and spaced Watch PRODUCT_NAME / CFBundleName matched to the spoken app name Removing updateAppShortcutParameters() from the watch App.init() Adding the com.apple.developer.siri entitlement to the watch target Phrase collision between phone and watch providers, a watch-only phrase also failed Siri domain collision on the trigger words, a neutral phrase also failed Intent identifier collisions between the phone and watch bundles A brand new, uniquely named intent with canonical syntax and no gating was also ignored Questions Is there a user-facing setting on watchOS, or in the paired iPhone's Watch app, to re-enable App Shortcuts for a specific app after the prompt has been declined? If not, is there any supported recovery short of erasing the watch and setting it up as new? Restoring from backup appears to bring the state back. Is there any API for an app to detect that its App Shortcuts opt-in is off, so it can show guidance instead of a silently dead feature? INPreferences.siriAuthorizationStatus() is not a valid proxy, it reports authorized on the affected device. Happy to provide a sysdiagnose with the App Intents logging profile, or either control project.
Replies
2
Boosts
0
Views
795
Activity
Aug ’26
Generation Error
So I'm having an issue with the FoundationModels framework but idk if this is just my feeling or not, the issue comes up after I updated my Mac into 26.6 the code was very simple actually: #Playground { let model = SystemLanguageModel.default let session = LanguageModelSession(model: model) print(model.availability) var query = "How to hide button" Task { do { let response = try await session.respond(to: query) print(response.content) } catch { print("\(error)") } } } the code works before I updated the version, but then after I updated the version it says: Error Domain=FoundationModels.LanguageModelSession.GenerationError Code=-1 "The operation couldn’t be completed. (FoundationModels.LanguageModelError error -1.)" UserInfo={NSMultipleUnderlyingErrorsKey=( "Error Domain=FoundationModels.LanguageModelError Code=-1 "(null)" UserInfo={NSMultipleUnderlyingErrorsKey=(\n "Error Domain=ModelManagerServices.ModelManagerError Code=1026 \"(null)\" UserInfo={NSMultipleUnderlyingErrorsKey=(\n)}"\n)}" ), NSLocalizedDescription=The operation couldn’t be completed. (FoundationModels.LanguageModelError error -1.)} this is runned in Xcode 26.6, additional information I have also coder 27 beta 4 installed in my Mac, is this problem occurring because the Xcode 26.6 and Xcode 27 beta 4?? can u guys help me
Replies
2
Boosts
0
Views
900
Activity
Aug ’26
Fused Metal Kernels for Linear Recurrences in MLX
I’ve been developing mlx-recurrence, a plug-in framework of fused Metal GPU kernels for linear recurrences on Apple silicon—roughly analogous to flash linear attention for MLX. Sequential recurrences are difficult for MLX to fuse automatically. Architectures such as state-space models, gated linear attention, and diagonal RNNs ordinarily require a loop across the sequence length. When that loop is implemented in Python, a sequence of length L can require L separate Python-to-Metal dispatches. These kernels instead execute the entire recurrence in a single Metal dispatch. The training path uses segment checkpointing with recomputation during the backward pass. In validated M3 Max tests, the checkpoint-and-recompute kernels reduced peak recurrent-state memory by approximately 12–18× at the kernel level and lowered total training peak memory from 23.88 GB to 10.34 GB. At the same batch size, end-to-end training throughput improved by roughly 1.4×, while individual fused forward-and-backward kernels ran approximately 1.5–1.9× faster than the original full-state implementations. Results will vary with recurrence type, sequence length, state dimensions, batch size, datatype, model architecture, and hardware. Current kernels: ssd_scan Mamba-2-style, head-wise SSD selective scan. Intended for Mamba-2 and other SSM hybrid architectures. State shape: [B, H, Dh, N] gla_scan Gated Linear Attention with a scalar forget gate and outer-product write. Intended for GLA and linear-attention hybrid architectures. State shape: [B, H, Dh, Dh] rglru_scan RG-LRU diagonal recurrence. Intended for Griffin and RecurrentGemma-style architectures. State shape: [B, D] rotlru_scan Rotational LRU using a complex-diagonal recurrence, a magnitude gate, and a per-step rotation of two-dimensional channel pairs. Intended for complex-LRU and S4-style oscillatory memory architectures. State shape: [B, D], represented as interleaved channel pairs. Each kernel is implemented as a self-contained plug-in on a shared chassis located at: mlx_recurrence._chassis The chassis provides: Segment checkpoint-and-recompute infrastructure Shape and argument validation VJP integration Forward and gradient parity-test helpers Common recurrence plug-in handling Adding another recurrence therefore requires implementing its Metal forward and backward source pair and connecting its VJP. The checkpointing, validation, and testing infrastructure does not need to be rebuilt for each operator. The original version 0.1 kernels remain available under: mlx_recurrence.legacy They are also re-exported at the package’s top level for backward compatibility. I’m interested in feedback from developers working with MLX or custom Metal compute kernels, particularly around: Preferred APIs for packaging reusable MLX Metal extensions Threadgroup and memory-layout strategies across Apple GPU generations Numerical stability expectations for long recurrent sequences Benchmarking fused scans against MLX-native implementations Additional recurrent operators that would be valuable to support I would also be interested to know whether others are developing similar fused recurrence primitives for MLX and whether a shared interface for these operations would be useful. My setup is a M3 MAX Macbook Pro with 36GB Ram and I am running on macOS 26.4.1 (25E253). https://github.com/D-CSIL/mlx-recurrence
Replies
0
Boosts
0
Views
307
Activity
Aug ’26
Xcode treats a `.llmasset` bundle as individual `.aimodel` files to compile, instead of copying it as-is
I have a Core AI model export — a bundle folder (.llmasset, containing multiple .aimodel subfolders plus metadata/tokenizer resources) — added to my app target as a folder reference. Rather than treating the bundle as one opaque resource and copying it into the app bundle as-is (the way .xcassets, .bundle, or any other folder reference behaves), Xcode reaches into it, finds the individual .aimodel subfolders, and adds each one to Compile Sources. When it compiles them there, it's for my build machine's specific chip only — I can't find any setting (Build Settings, scheme, target picker) to compile for multiple architectures/platforms, the way a universal binary would work. Question: Is there a way to make Xcode treat a .llmasset bundle as an atomic resource — copied wholesale, not decomposed into individual .aimodel compile targets? Or is reaching into the bundle and AOT-compiling its components for the active build architecture the intended behavior here, and if so, what's the recommended way to make sure the result works across the actual range of devices the app ships to?
Replies
1
Boosts
0
Views
470
Activity
Aug ’26