#511

Result cards for Trip.com, Taobao, Dianping, Uber Eats, journeys with changes and any app

OpenZhehanZhangwants to merge intofrom

Summary of changes

finish_task's result card can now be drawn for more than 12306's train list. New kinds, each drawn the way its app lists results, using native SwiftUI and the card rules from design/result-cards-reference: one selected item, one primary button, grey facts.

  • ctrip.flights, ctrip.hotels, ctrip.rooms and ctrip.fares for Trip.com (携程) flights, hotels, a hotel's room types and a flight's fares.
  • taobao.products, dianping.places and ubereats.items.
  • 12306.transfers for 12306's 中转 list. It shows the first departure and last arrival, the change station in a pill under the journey's length, what the change takes in orange (车内换座) and the seats left on each leg (1程, 2程).
  • Connections in ctrip.flights. Each connection is listed under its flight with the city, the times between, the layover and 不同航站楼转机.
  • list, a neutral look for any app without its own, so a new app is never text only.

All the new looks draw their rows through one shared list (ResultCardList) with the app's icon and name, a heading and the selection mark. Journeys with changes share one legs field: number, from, to, depart, arrive, wait, change and seats per leg.

Item pictures. An item can give image: [x, y, width, height], the box of its picture in the latest screenshot's points. When the task finishes, PokiCore's itemPictures takes the latest screenshot (from the last two minutes, not a split view pane). It matches each box to the image element it overlaps (IoU at least 0.6) and leaves out a picture that the screen's edge or a bar cuts. The app then cuts the rest, keeps them with the card (at most 180 px JPEG each) and shows them at the start of the row. Pictures are only ever cut from the phone's screen and never taken from the model.

Motivation (Closes #N)

Related to #484 (result cards beyond 12306). I tested this against real screens: 11 cases from 16 screenshots of the apps above, run on GPT-6.1 Terra and Sol, formats A and B, 3 runs each.

  • Values: across 132 cards no value was made up, and 113 matched the answer key exactly.
  • Missing looks: 10 of those 11 screens had no look to draw them on main.
  • Pictures: both models boxed all 25 item pictures on 6 real screens tightly, with the same boxes on every run.

Test plan

  • PokiCore: swift test --filter ItemPictureTests passes on Linux (Swift 6.2). It covers matching a box to its image, dropping pictures cut by the screen's edge or a bar, and dropping boxes that are too small.
  • App: new PokiTests/ResultCardLookTests.swift, which covers each kind's required fields, legs, the list look, the finish_task schema, the image box, and pictures surviving a saved card. The app code has not been compiled yet. The first build will tell whether it does.
  • To test on a phone, ask Poki to pick from a list in each app. For example: 12306 with a change of train, a Ctrip flight with connections, Ctrip hotels, Taobao search, Dianping nearby, and an Uber Eats store. Check that the card matches the app, the selection and buttons follow your tap, and pictures appear only for items whose picture was fully on screen.

Screenshots or screen recording

Not yet. The app hasn't been built, so screenshots come with the first build.

@JosssphZhou could you build and test this on a phone?

Activity

3
JosssphZhou2d ago

Thanks for building these cards from real examples. I could select a second room and saw the hotel name disappear. Please keep the hotel name across selections and tie each thumbnail to the screenshot that supplied its coordinates.