Back to work

02 / HARNESS · TOOL USE

DeviceAgent Arena

Evaluate the whole trajectory.

A testbed for Chinese tool-use agents, where actions change the next state. Check both the outcome and the path taken.

Python / Stateful tasks / Reproducibility

The problem

Choosing the right function is only part of tool use. An agent also needs to handle state changes, missing information, failures, and safe stopping conditions over a sequence of actions.

The approach

A bounded harness runs tasks in a deterministic text simulator with state transitions and fault injection. Graders inspect the final world state and the complete action trajectory. The project also explores small-model training and compound execution.

Evidence & scope

Experiment reports and a reproduction guide document the setup and its limits. The development suite is small and shares instruction templates with training; its results describe that environment and do not establish production performance.