How are you iterating on Foundation Models prompts before building the app workflow?

Question

svgeek OP

Created 2w

Replies 4

Boosts 0

Participants 2

While building with Apple's Foundation Models, I kept running into a workflow problem before the app code itself.

The hard part was not only calling LanguageModelSession. It was figuring out the shape of the interaction:

What should be in the system prompt? What should stay in the user input? What output is actually usable by the app? How much instruction is too much? How do I test the same prompt repeatedly without creating another small Xcode project?

I ended up building a small macOS tool for myself, LocalLM Lab, mainly to speed up that loop. The first use case was a Prompt Playground: system prompt, user input, model output, and a repeatable way to compare results before moving the workflow into app code.

The current version also experiments with connector-style context, such as system clock, weather, reminders/calendar, contacts, and a scoped filesystem folder. That has made the prompt design problem more interesting, because the question becomes: what context should the model see, and how should the app frame that context so the output is useful?

I am curious how other developers are handling this while building with Foundation Models.

Are you mostly iterating inside Xcode playgrounds? Are you building small internal test harnesses? Are you separating system prompts and user inputs during testing? How are you evaluating whether the output is reliable enough for the app workflow? For reference, this is the tool I have been using for my own experiments:

https://thisbrain.ai/locallm

I would be especially interested in any patterns people have found for designing and testing prompts before committing them to app code.

Answer 1

Apple Designer OP

Apple

2w

That's a fun idea, thanks for sharing!

For simple prompts, I like to try out the new fm command line tool in Terminal on macOS 27+, but for more complex apps, yeah, I like experimenting in Xcode #Playground or having an agent in Xcode build me a mini test harness app. 🙂

Answer 2

svgeek OP

1w

Small(-ish) update relevant to the context-budget part of this thread: I ended up building MCP client support into LocalLM Lab (v0.4), and it turned the "what fits in the context window" question from the original post into something very concrete.

MCP servers advertise their tools with full JSON schemas, and those schemas count against the same ~4096-token budget as everything else the model sees. Some servers are cheap (a couple hundred tokens for everything); others are not. For example, Linear's full tool list alone would blow the budget on its own. So the UI ended up being: every tool starts disabled per server, and you enable only what a given task needs, one or two at a time. This covers both privacy+security in a dev environment.

It's a more literal version of the same problem this thread was originally about - deciding what the model actually needs to see versus what's just available. One upside of building against this specific constraint: since the model's already local and free to call, there's no API cost or sandbox setup involved in actually watching tool discovery and tool calls happen against a real server - useful if you're trying to understand the protocol itself, not just ship a feature with it.

I am curious whether anyone else doing tool-calling with LanguageModelSession has run into the same tradeoff, and how you're deciding what to expose per request versus keeping available but unused.

Answer 3

svgeek OP

1w

Oh and I forgot to include the writeup on the different MCP servers I have tested with: https://thisbrain.ai/locallm/mcp-servers.html

Answer 4

svgeek OP

1d

Another update relevant to this thread's original question about iterating on prompt/context shape before writing app code: v0.6 adds localai-cli, a small subprocess binary for calling the on-device model straight from your own Swift (or Python) code once you've got a prompt shape you're happy with.

Mechanically it's about as simple as it gets: your app spawns the binary, writes one JSON request (system prompt + input, or a messages array for multi-turn) to stdin, reads one JSON response from stdout. No HTTP server and no client library. This is useful if you've been iterating in Prompt Playground and want to move that exact prompt into a real Xcode project without wiring up LanguageModelSession and the connector/MCP plumbing yourself.

It's config-scoped to whatever the user already granted through the app's own settings ( connectors and MCP tools included) so a request outside that set is rejected before the model runs at all. Omitting the field grants nothing.

Being upfront: this isn't a packaged SDK for shipping. LocalLM Lab has to be running for calls to work, since they relay through its background process. It's meant for the iterate-then-verify step this thread was originally about, not for distributing inside another app yet.

There's a fuller worked example on the page now too. "Plate Today" combines three connectors (Calendar, Reminders, System Clock) with an MCP tool call (Todoist's find-tasks-by-date) in a single request, and reads localai-config.json itself first to confirm everything's actually enabled before ever invoking the model. Useful reference if you're weighing how much validation to push into your own code versus relying on localai-cli's own rejection behavior. The example does both, and the preflight is purely about not spending a model call on a request you already know will fail.

Interface + Swift/Python examples: thisbrain.ai/locallm/cli.html