microsoft/amplifier-bundle-ios-tester

microsoft/amplifier-bundle-ios-tester

Description: iOS tester bundle for the Amplifier project

Language: Python

License: MIT

Stars: 0

Forks: 0

Open issues: 0

Created: 2026-08-07T22:17:12Z

Pushed: 2026-08-15T04:26:41Z

Default branch: main

Fork: no

Archived: no

README:

amplifier-bundle-ios-tester

Let an agent actually see and drive the iOS screen — so UI fixes stop shipping "verified" and arriving broken.

Unit tests pass. Server-side curl passes. The user opens the app and the fix is broken. The failure is always at the render/interaction layer, and nothing in the loop has ever looked at the screen. This bundle closes that gap.

The Load-Bearing Rule

The accessibility tree is the sensor. The screenshot is for judgment, never for targeting.

Measured live on a booted simulator — same screenshot, same element (the "General" row in Settings):

| Source | Pixels | Center | |---|---|---| | ui_dump (the accessibility tree) | [48,1132][1131,1288] | (590, 1210) | | VLM reading the PNG | [36,1000][1143,1160] | (590, 1080) |

Delta dy −130px — the VLM-derived center lands outside the real row, on the row above.

And the miss is silent. iOS has no concept of "you tapped nothing": no error, no exception, and often a screenshot that still looks right. The dangerous case is landing on a *neighbouring* element — adjacent settings rows sit 156px apart, so a 130px error reliably selects the wrong row. You navigate into the wrong screen, assert against it, and report success.

So: vision answers *"does this look right / what state am I in / is anything clipped"*. Coordinates always come from `ui_dump`.

The iOS-Only Trap: Points vs Pixels

Android has one coordinate space. iOS has two, and they differ by a factor of 3.

accessibility tree: 393 x 852 (POINTS)
screenshot: 1179 x 2556 (PIXELS)
scale: 3.0

Mixing them is a silent 3× coordinate error — larger than the VLM miss above, and equally invisible. It is also a phantom-bug generator: an element that looks "wildly misplaced" when you compare its frame against a screenshot is almost always a unit error, not a layout defect.

The tool owns every conversion and never hands you a number whose unit is ambiguous:

node = {
"label": "General",
"frame_points": {"x": 16.0, "y": 377.33, "w": 361.0, "h": 52.0},
"frame_pixels": {"x": 48, "y": 1132, "w": 1083, "h": 156},
"center_points": [196.5, 403.33],
"center_pixels": [590, 1210],
}

Agents use the labeled field. They never convert by hand, and they state the unit in every geometric claim they report.

Three Tiers, Honestly Labeled

Feasibility here is not binary. It splits by what you are willing to pay for.

| Tier | Setup cost | Can SEE | Can TAP | Status | |---|---|---|---|---| | Simulator | Xcode only | labels + frames | yes | proven | | Device — free | cable + Developer Mode | screenshot, element list, syslog | no | proven | | Device — WebDriverAgent | + Apple Developer account, signing | yes | yes | unproven, not implemented |

The middle tier is real and worth shipping: device identity, app inventory, live syslog, and screenshots with no Apple Developer account and no signing. It simply cannot drive the UI, because the element list carries no geometry — this is a complete element, verbatim:

{ "caption": "100% battery power, Charging, ...",
"estimated_uid": "06000000-...",
"platform_identifier": "21000000C048...",
"spoken_description": "..." }

Full key set: caption, estimated_uid, platform_identifier, spoken_description. Grepping the whole payload for frame|rect|bounds|x|y|width|height returns zero.

So `tap` on `backend="device"` refuses rather than guessing, and names WebDriverAgent as the path that would provide geometry. A read-only tier that honestly says "I cannot tap" is worth more than a tapping tier that guesses.

What Was Proven, and How

Simulator (macOS 26.6, Xcode 26.6, arm64, over bare SSH):

| Step | Evidence | |---|---| | Boot headless over SSH | 2.7s to Booted — the single biggest feasibility risk, cleared | | Screenshot | 1179×2556 PNG via simctl io screenshot | | Element tree + frames | 33 labeled elements; General → {{16, 377.33}, {361, 52}} | | Selector-resolved tap | axe tap --label "General" → resolved (196.5, 403.3) → tree went 32 → 73 elements |

Physical device (iPhone X, iPhone10,6, iOS 16.7.16, arm64):

| Step | Evidence | |---|---| | Pair + identify | UDID, model, build 20H392 | | Developer Mode | enabled programmatically — confirmed true by two independent tools | | DDI mount | Xcode 26 ships no 16.7 image; the 16.4 image mounted on 16.7 anyway | | Screenshot | 1125×2436 PNG off the real device | | Element list | works — but no geometry |

How It Works

A single ios_inspector tool wraps simctl, axe, and pymobiledevice3 with a selector-first contract — the safe path is the default path:

• tap cannot fire without resolving a selector against a fresh accessibility dump
• every geometric field is returned in both coordinate spaces, explicitly labeled
• type_text asserts focus before typing and asserts readback after
• wait_for polls the tree; there are no bare sleeps anywhere
• tap_xy (raw coordinates) is named to be conspicuous and warns in its own result
• tap on the free device tier refuses, because there is no geometry to resolve against

An agent *told* to follow these rules skips them at turn 40 of a long run. A tool that structurally *cannot* skip them does not.

Quick Start

Installation

Add as an app bundle (recommended):

amplifier bundle add git+https://github.com/microsoft/amplifier-bundle-ios-tester@main#subdirectory=behaviors/ios-tester.yaml --app

Compose into another bundle:

includes:
- bundle: git+https://github.com/microsoft/amplifier-bundle-ios-tester@main#subdirectory=behaviors/ios-tester.yaml
as: ios-tester

Prerequisites

Start by asking the bundle. doctor takes no parameters, never errors, and tells you everything that is wrong in one call:

report = ios_inspector(operation="doctor")
# report["ready"] — false if any check failed
# report["checks"] — [{name, status: ok|warn|fail, detail, remediation}, ...]
# report["summary"] — what to fix first

It checks Xcode / DEVELOPER_DIR, simctl, axe, pymobiledevice3, available runtimes,...