#463

Routine fixes on the way and an optional decision model

DraftZhehanZhangwants to merge intofrom

Summary of changes

Stacked on #490, which now holds the one-call paths, apps by name and the quicker read that used to be here. What is left is the part that touches how routines are remembered, so it can land together with the memory rework.

  • Fixes on the way. A routine or path that meets a prompt or a renamed button first applies what the routine remembers from an earlier fix, with no model.
  • Intent. Any action may carry intent: "why". It is kept with the recorded step and is what a fix is decided from.
  • Decision model (optional). It is picked on Settings › Models beside the other models: Off, Apple Intelligence where the iPhone has it, or a model of the owner's own provider (Cloudflare Workers AI, TypeSafe, OpenAI's Decisions API, or a custom address). The In Use card shows Decision in place of Dictation. There is no switch: the model is in use once picked. It is asked only where a step would otherwise stop, acts only on a clear answer, and never taps anything that needs approval. With a model picked, a path target may be {find: "what it is"} and a step may carry check.
    • Apple Intelligence decides on the phone and sends nothing out. It answers one prompt with JSON that is read as strictly as a decision model's answer, and gets 8 s where an API gets 2.5 s. Its confidence is the model's own word, not a measured probability.
    • A provider's request goes from the iPhone straight to that provider, as text only, with no screenshot and nothing from a secure field. OpenAI uses the key entered on the providers' page, or the OpenAI API key already saved on the Models page. A ChatGPT sign-in is not used: OpenAI answers 401 to its token on the Decisions API.

Downloaded models on the phone as the decision model are in #487, stacked on this one.

Where a fix is stored

Answering the question in the comments: fixes are not stored per app. They are stored inside the routine, on the step they belong to, in RoutineStep:

  • also: other forms the step's target took in a run that worked (a renamed button), at most 3.
  • clears: what was in the way of this step before and was closed (a prompt's Not Now), at most 4.
  • Routine.fixes counts steps fixed over all runs.

So a fix is per app only as far as its routine has an app. Nothing is shared between two routines of the same app. If routines become per-app skills, clears is the part that would make sense at app level (the same prompt gets in the way of every routine in that app), and also belongs with the step.

Motivation

A routine stopped at the first prompt or renamed button and handed the whole task back to the agent's model.

Test plan

  • swift test --filter 'DeciderTests|PathTests|RoutineTests' for PokiCore on Linux at efd69f4: 44 tests, 0 failures. The whole suite passed at 328 tests one commit earlier (cff589f, before the restack, which moved files and changed one line). The OpenAI request and answer shapes are tested against the examples in OpenAI's Decisions guide.
  • On a physical iPhone: Check Connection against OpenAI with a ChatGPT sign-in answered 401, which is why that path was removed.
  • Not yet done: the Models page change at the head is not compiled, no successful call to a real decision model provider, and Apple Intelligence has not been tried as the decision model.

Screenshots or screen recording

To come with the first build of this head.

  • I removed passcodes, tokens, and private screen contents from this PR.
  • I release this contribution under the AGPL-3.0-only license.

Activity

33
JosssphZhou4d ago

Thanks for this one. We're about to restructure memory on our side: routines become per-app skills, and the harness recalls memory by the current app (the app filter at Harness.swift:48 is written but never passed). That touches Routines.swift and Harness.swift, the same files this PR rewrites, so we'd like to build on top of your work instead of colliding with it.

Two quick questions:

  1. Does Repair.swift ("remember the last fix") store repairs per app? If so, we'll treat it as the first per-app skill memory and extend it.
  2. Could you split the one-call paths and launching apps by name into their own PR? Those look ready to go on their own, and the routine and repair part can then land together with the memory work.

No rush on the draft otherwise.

This pull request is a draft

Merging is unavailable until it is marked ready for review.