Findings of EMNLP 2026

Agent-Software Interaction Layer

Replacing Screenshot-and-Click with Structured State and Semantic Actions

Rui Xie1  ·  Lu Chen1

1Shanghai Jiao Tong University

sharryXR@sjtu.edu.cn  ·  chenlusz@sjtu.edu.cn

Paper Code Models Benchmark Training Data BibTeX

TL;DR: ASIL replaces visual motor control with structured software state and semantic actions. Closed models score above 80 overall, while repaired screenshot control reaches 6.6 and 26.6 on the same 380 tasks.

One task, two interfaces

What changes when the interface changes?

Follow the same LibreOffice payroll task from instruction to outcome. GUI control operates through pixels and motor events; ASIL operates through task-relevant state and one semantic action.

Task

Create a payroll sheet with nine specified columns and exactly ten complete employee rows. Net Pay must equal Base Salary + Position Allowance + Performance Bonus - Insurance - Tax.

GUI: Screenshot + Clicksame initial file
Initial LibreOffice Calc payroll task before GUI interaction
Receives the task instruction and a screenshot of the visible spreadsheet surface.
ASIL: State + Semantic Actionsame initial file
The same initial LibreOffice Calc payroll task before ASIL interaction
Receives the identical task, initial artifact, and final-state evaluator.
15
real software
applications
380
shared benchmark
tasks
<5
average ASIL actions
per task
3
access-path
families

Abstract

An agent-native interface for real software

Screenshot-and-click is an inefficient interface for software-operating agents: screenshots are state-incomplete, while GUI actions are brittle, semantically weak, and poorly matched to long-horizon planning.

ASIL exposes software through structured JSON observations and code-executable semantic actions, using the deepest feasible open access path for each application. The implementation spans 15 applications, 300 single-application tasks, and 80 multi-application tasks. Closed models exceed 80 overall while executing fewer than five ASIL actions per task; repaired 50-step screenshot control reaches 6.6 and 26.6 for GPT-5.4 and sonnet4.6. The structured modality also supports compact, verifiable data for SFT and evaluator-backed on-policy RL.

The interface mismatch

A software agent should not have to imitate a human hand

Modern agents already reason in text and code, yet graphical software forces them through a presentation layer designed for people. One semantic intent becomes a sequence of clicks, drags, key presses, and repeated visual grounding.

ASIL changes the observation-action contract while preserving the task, initial artifact, and software-aware evaluator.

Screenshot-and-click

pixels -> coordinates -> motor events
Visible projection, long action horizon, layout-sensitive execution.

ASIL contract

structured state -> semantic action -> check
Task-relevant objects, executable intent, replayable validation.

Method

Use the deepest feasible software access path

ASIL normalizes reviewed file parsers, native scripts, structured commands, and service APIs into one agent-facing observation-action protocol.

ASIL overview contrasting screenshot-and-click GUI control with structured state and semantic actions
GUI and ASIL expose the same software task through different contracts. ASIL reads task-relevant state, executes a semantic operation through the application's deepest stable access path, and checks the resulting software state.

File-backed

Parse open document and project formats, then apply reviewed structural mutations.

6 implementations: Inkscape, LibreOffice, draw.io, JupyterLab

Native script

Use the application's own scripting environment or editable project graph.

4 implementations: Blender, GIMP, Kdenlive, Audacity

Service / API

Bridge REST, WebSocket, filesystem, or desktop-service state into the shared contract.

5 implementations: OBS, Gitea, code-server, Thunderbird, Nautilus

Semi-automatic ASILization

Turn reviewed interface evidence into an auditable adapter candidate

The public onboarding framework automates repeatable assembly and validation after a human has identified a trustworthy observation and action path.

  1. Qualify

    Classify direct, bridge-assisted, or out-of-scope access from cited evidence.

  2. Generate

    Compile a typed interface plan into an extension bundle and action schema.

  3. Audit

    Fail closed on invalid evidence, permissions, transport, or operation contracts.

  4. Probe

    Run observation-first host and Docker checks with explicit action gates.

  5. Report

    Recompute hashes, provenance, audit results, and validated state changes.

Measured Gitea onboarding

One GPT-5.4 call compiled a reviewed 97-line API profile into one observation view and two semantic actions.

24.8 seconds0 audit errors3/3 host probes3/3 Docker probes

Human review remains explicit

Interface discovery, parser or native-bridge implementation, evaluator design, GUI synchronization, rendering, and final benchmark acceptance are not delegated to the generator.

Read the extension architecture

Results

The interface changes the operating regime

Strict success on the shared 380-task benchmark. GUI rows are shown only where the accepted paper reports matched repaired controls.

Main benchmark overall score (%). Budgets are shown as 15max or 50max; GUI rows use the repaired native-computer-use runtime.
ModelInterfaceBudgetOverall (380)
Kimi K2.5ASIL15max84.8
GPT-5.4ASIL15max81.6
GPT-5.4Screenshot GUI50max6.6
sonnet4.6ASIL15max81.2
sonnet4.6Screenshot GUI50max26.6
sonnet4.6Screenshot GUI15max17.9
Qwen3.6-plusASIL15max81.1
Qwen3.5-27BASIL15max75.8
ASIL exceeds LibreOffice's UNO API by 28-38 strict points on matched tasks, while matching draw.io's MCP content contract for GPT-5.4. The result is an interface claim, not a claim that every ASIL access path dominates every mature native interface.

Training under ASIL

Structured traces make small-scale post-training effective

The same evaluator filters demonstrations, supplies rollout rewards, and evaluates the resulting checkpoints on all 380 tasks.

Qwen3.5-2B

Base
58.0
SFT
72.1
RL
74.4

Qwen3.5-9B

Base
66.6
SFT
80.4
RL
82.2

Application coverage

One contract across creative, office, developer, and desktop software

Representative real software surfaces selected from evaluated benchmark runs. Open any image to inspect it at full size.

Representative Blender benchmark interface
Blender
Representative GIMP benchmark interface
GIMP
Representative Inkscape benchmark interface
Inkscape
Representative Kdenlive benchmark interface
Kdenlive
Representative Audacity benchmark interface
Audacity
Representative LibreOffice Calc benchmark interface
Calc
Representative LibreOffice Impress benchmark interface
Impress
Representative LibreOffice Writer benchmark interface
Writer
Representative draw.io benchmark interface
Draw.io
Representative code-server benchmark interface
Code
Representative JupyterLab benchmark interface
JupyterLab
Representative Gitea benchmark interface
Gitea
Representative Thunderbird benchmark interface
Thunderbird
Representative Nautilus benchmark interface
Nautilus
Representative OBS Studio benchmark interface
OBS Studio
Representative multi-application benchmark workflow
Multi-App

The gallery illustrates benchmark coverage; it does not represent a release of per-task evaluation trajectories.

Open resources

Build, train, and evaluate with ASIL

Code, released checkpoints, benchmark tasks, and training data are maintained as separate public resources.

Code

Runtime protocol, 15 application adapters, evaluators, baselines, containers, and the assisted onboarding framework.

Apache-2.0 / inference + evaluation View on GitHub

Models

Selected Qwen3.5-2B and Qwen3.5-9B SFT and RL checkpoints reported in the paper.

2B + 9B / SFT + RL Browse models

Benchmark

Task definitions for the 380-task main benchmark, the 80-task hard suite, and the easy60 comparison band.

300 single-app + 80 multi-app Browse benchmark Runtime images

Training Data

Prepared supervised and rollout-curriculum data used for the released ASIL post-training experiments.

SFT data + RL curriculum Browse training data

Citation

BibTeX

@inproceedings{xie2026asil,
  title     = {{ASIL}: Replacing Screenshot-and-Click with
               Structured State and Semantic Actions},
  author    = {Xie, Rui and Chen, Lu},
  booktitle = {Findings of the Association for Computational
               Linguistics: EMNLP 2026},
  year      = {2026}
}