AnyDecisionModel: choose a review threshold before trusting confidence

October 29, 2026. AnyDecisionModel is a new Swift package for applications that need a choice or score instead of a paragraph. It can return typed decisions with probabilities, but those numbers still need an application-specific interpretation before they control an action.

In today’s announcement, developer Mattt introduces AnyDecisionModel alongside AnySpeechRecognizer. The former supports yes-or-no questions, choices and rating scales through local and hosted backends; the latter offers alternative speech engines behind familiar Apple-style APIs. For decision results, the announcement distinguishes confidence based on probability concentration from correctness, and recommends separate datasets for calibration and evaluation.

A typed answer solves a formatting problem

Suppose a fictional document app assigns incoming notes to “question,” “task” or “reference.” Receiving one defined label is easier to handle than parsing an unconstrained sentence. It does not establish that the label fits the note. A perfectly valid “task” value can still be the wrong classification.

Keep the benefits separate when planning the feature. One acceptance check asks whether the response fits the application’s allowed values. Another asks whether a reviewer agrees with the choice. A third asks whether the application should act automatically or offer a correction first.

Choose a review threshold with counted outcomes

The following is an original worked example with invented numbers, not a measurement of this package or any model. Imagine an evaluation set of 100 labeled notes. Your proposed confidence threshold accepts 60 notes automatically, and five of those labels are wrong.

That gives 60% automatic coverage and about 8.3% error among the automatically accepted notes: five divided by 60. Dividing five by all 100 notes would produce 5%, but would answer a different question. Users exposed to an automatic label care about the mistakes within that accepted group.

Now suppose a stricter threshold accepts only 35 notes, with one error. The accepted-group error is about 2.9%, while 65 notes need review. Neither threshold is inherently the right one. The choice depends on how much incorrect labeling disrupts the app and how much review users will tolerate.

Present both quantities together when deciding. “More confident results” is less useful than “35 of 100 accepted, one wrong.” If there are only a handful of accepted examples, collect more before treating the estimate as stable.

Build the evaluation before adjusting confidence

1. Define each label. Write examples and boundary cases. A note can mention a task without asking anyone to do it.
2. Prepare a calibration set. Use it for adjustments to the probability mapping or review rule. Keep a separate evaluation set untouched until the rule is chosen.
3. Inspect disagreement. If reviewers disagree about the intended label, resolve the labeling policy rather than expecting a model score to settle it.
4. Report by useful subgroup. For this app, compare short notes, long notes and mixed-purpose notes. An overall number can hide the cases users most often correct.
5. Keep a correction path. Let users change the label and preserve enough context to diagnose recurring mistakes.

These are proposed product checks, not claims that the package performs calibration automatically for your data. The author’s calibration discussion also explains an option-order debiasing feature in next-token backends, which adds evaluations by rotating choices. That is another configuration to record when comparing results.

Local execution needs a complete app check

The announcement says both packages can download models and run on device. It also says its speech measurements were made on a Mac, with iPhone testing still outstanding. Treat those published measurements as scoped evidence, rather than an iPhone performance promise.

For a local note classifier, measure the complete first-use experience: download, model load, first decision and subsequent decisions. Keep those timings distinct. A fast decision after loading does not tell a user how long the feature takes to become available after installation.

If speech later supplies the note text, preserve the original transcript during testing so you can identify whether an incorrect label began with a transcription error. Our recorded-audio evaluation guide gives examples of errors that can change meaning.

AnyDecisionModel’s useful contribution is a structured way to ask bounded questions. The application still needs to decide what the answers justify. Start with reversible suggestions and measured review coverage before allowing a confidence score to trigger consequential actions.

This article reviews the release documentation and proposes an evaluation. We have not compiled or benchmarked the packages.

Illustrative photograph: reading workspace by Yen Vu / Unsplash, under the Unsplash License. This is not a screenshot of the package or an Apple product demonstration.

Sources

• Mattt: Introducing AnySpeechRecognizer and AnyDecisionModel — October 29, 2026; package behavior, confidence, calibration and testing scope. The numerical example and application workflow above are original analysis.