Local inference is often reduced to one appealing claim: the prompt never leaves the device. That property matters, but it does not tell us whether the model is useful, whether intermediate data is retained, whether the device overheats, or whether an unavailable local model quietly sends the task to a cloud fallback.
We therefore treat local-first AI as a system-level question. The model, runtime, operating system, hardware, application, storage policy, and network behavior form one evaluation unit. Changing any of them can change the privacy or usability result.
1. Draw the privacy boundary first
Before measuring speed, document what information exists and where it can travel. A useful boundary diagram should include the original user input, retrieved documents, prompt templates, model weights, embeddings, generated output, logs, crash reports, analytics, backups, and optional cloud services.
For each artifact, answer four questions:
- Where is it created?
- Where is it stored, and for how long?
- Which process or person can read it?
- Can any network request contain it, directly or indirectly?
A system can perform model inference locally while still uploading prompts through telemetry or syncing generated text through a general backup service. “Local model” and “private workflow” are related but not interchangeable descriptions.
2. Describe the complete test unit
Results are difficult to interpret without a reproducible configuration. At minimum, record:
- Device model, operating system, available memory, processor, GPU or accelerator, and power mode.
- Model name and version, parameter count, quantization, context limit, and weight source.
- Runtime and version, execution backend, thread count, batch settings, and sampling parameters.
- Application version, prompt template, tool configuration, retrieval settings, and network policy.
- Whether the run is cold or warm and whether model weights are already cached.
The goal is not exhaustive inventory for its own sake. It is to make clear which system produced the observation and which changes could plausibly explain a different result.
3. Measure six dimensions together
| Dimension | Question | Example measurement |
|---|---|---|
| Privacy | Did sensitive data stay inside the declared boundary? | Captured network traffic, retained files, telemetry inspection |
| Quality | Did the output accomplish the user’s task? | Task-specific rubric, error classes, human preference |
| Latency | How long did the experience feel blocked? | Load time, time to first token, total completion time |
| Resources | Can the target device sustain the workload? | Peak memory, storage, CPU/GPU use, thermal throttling |
| Energy | What does repeated use cost the device? | Battery change or energy consumed per task |
| Recovery | What happens when local execution cannot finish? | Cancellation, resume, fallback visibility, data handling |
Tokens per second belongs inside latency, but it is not a user outcome by itself. Loading a model for forty seconds and then generating quickly may be worse than slower generation from an already available model. The right measurement follows the complete task.
4. Use tasks that represent the product
A generic language benchmark can help compare models, but it may not predict whether a local feature works. A browser summarizer, private document assistant, coding tool, and game agent have different inputs, acceptable delays, and failure costs.
Build a small task suite from realistic use cases. Include short and long inputs, ordinary and adversarial cases, expected refusals, interrupted runs, low-memory conditions, and at least one case where the correct behavior is to request help rather than guess.
Keep the suite stable long enough to compare versions. When a test changes, version it. Otherwise an apparent model improvement may simply be an easier prompt.
5. Test the deployment ladder
“Local versus cloud” is often presented as a binary decision. A product may instead use a deliberate ladder:
- Device-only: the task and all supporting data remain on the user’s device.
- Local network: inference runs on hardware the user controls nearby.
- User-chosen remote: the user explicitly selects a provider and understands what leaves the device.
- No execution: if the privacy or quality requirement cannot be met, the product declines.
Every transition should be visible and testable. Silent fallback undermines the strongest reason to choose local inference in the first place.
6. Publish distributions and failures
Report medians together with slower runs, not only the best attempt. Long tails matter when a device is under memory pressure or competing with foreground work. For quality, publish error categories and representative examples rather than compressing everything into one average.
Failures are especially useful: corrupted model downloads, context overflow, out-of-memory termination, thermal throttling, tool timeouts, and unsupported hardware reveal whether the application can protect the user’s work.
A minimum useful report
A concise local-inference report can still be rigorous. It should include the privacy boundary, complete configuration, task suite, measurement definitions, distributions, representative failures, fallback behavior, and limitations. Raw traces or scripts should accompany the report when they can be shared safely.
Private local inference succeeds when the system preserves the promised boundary and completes a real task acceptably—not when a model merely produces tokens without a network connection.
Open questions
- How should a product communicate a changing privacy boundary without exhausting the user?
- Which quality regressions are acceptable in exchange for local control?
- How can benchmarks include background load and long-term thermal behavior without becoming impractically expensive?
- What artifacts make a local-inference claim independently auditable?