#465

Learn the usual music app from Screen History

OpenJosssphZhouwants to merge intofrom

What this changes

Poki can learn one habit from Screen History: the owner's usual music app. When the owner asks for a song and names no app, Poki opens that app.

  • Off by default. A new switch, Learn Habits from Screen History, sits under Screen History in Settings. It stays off after an update and needs Screen History on.
  • Three uses in a week, then the model decides. PokiCore sums up the owner's own uses of each music app from Screen History in the past week: how many uses, on which days and at what time of day. Reads within 30 minutes count as one use, and uses while an agent had the app open are left out. Only an app used at least three times in that week can be a habit; the number is a constant in code (MusicUsageSummary.minUses), not in the prompt. The chat's model reads the summary of the apps that qualify with a description of what a habit is, and decides whether there is one. It writes a short piece of evidence. When no app qualifies, the model is not asked and nothing is learned. The model is asked before a music request that names no app and when the Memory page opens, and again only when the summary changes. No screen text goes into the summary.
  • The owner's words come first. "From now on, use X" is always saved as a note, whatever app it names, including Apple Music, Migu, or an app Poki has no name for. Such a note, or an app named in the request, keeps the learned app out of the instructions. Only saving a routine is refused while the learned app is in the task.
  • Memory page. A section, Learned from Screen History, shows "Usual music app: …" marked Inferred automatically, with the model's evidence and the day it was learned, and a Forget button. The Memory row in Settings counts it.
  • Forgetting is final. Turning off either switch, Forget, or deleting part or all of Screen History drops what was learned and writes a watermark. Reads before it are not summed up again, including after a restart. The owner's own notes, such as "以后都用 QQ 音乐", are not learned from Screen History and stay.

This branch includes the commits of #435, which is up to date with main. Once #435 merges, only this branch's own commits remain. Music app names live in this change, so KnownApps is not touched.

How you tested it

  • ScreenHabitTests in PokiCore, 7 cases:

    • a hundred reads of one page are one use;
    • three uses in a week reach the model with their days and times;
    • fewer than three uses in a week, or older uses, are not offered;
    • Poki's own launches are not the owner's uses;
    • uses before Forget are left out;
    • an app left out of Screen History is left out;
    • only an app from the summary is taken from the model's reply.
  • ChatHarnessTests drive the chat against a local fake model endpoint.

    • Three uses in a week become the habit when the model says so, and the task gets the model's evidence.
    • Two uses in a week, with older ones, make no habit, and the model is not asked. Failed before this change.
    • Deleting all of Screen History forgets the learned app, so the next music request does not use it, and keeps the owner's "以后都用QQ音乐放歌" note. Failed before this change.
    • Turning Screen History off and on again does not bring the learned app back.
    • Six uses on three days make no habit when the model says there is none.
    • "以后都用苹果音乐放歌" and "以后都用咪咕音乐放歌" are saved while NetEase Cloud Music is learned, and keep it out of the next music request.
  • ScreenHistoryTests in the app, 12 cases. They include that a named app or a saved note keeps the learned app out, that deleting part of history forgets it, and that turning learning off forgets it.

  • App tests ran on the iOS Simulator by injecting XCTest into the app, because xcodebuild test hung before the runner connected on this Mac. Not yet checked on a physical iPhone or with a real model.

  • I removed passcodes, tokens, and private screen contents from this PR.

  • I release this contribution under the AGPL-3.0-only license.

🤖 Generated with Claude Code

Activity

19
JosssphZhou2d ago

@ZhehanZhang Screen History is your area, so we're pausing this PR to avoid overlapping with #488 (both touch HistoryCapture, HistoryStore and HistoryView). We won't push more changes here. If the habit idea (learning the usual music app from Screen History) fits your plan, feel free to fold it into your work or close this one, whichever you prefer.