02 / HARNESS · TOOL USE
DeviceAgent Arena
Evaluate the whole trajectory.
A testbed for Chinese tool-use agents, where actions change the next state. Check both the outcome and the path taken.
The problem
Choosing the right function is only part of tool use. An agent also needs to handle state changes, missing information, failures, and safe stopping conditions over a sequence of actions.
The approach
A bounded harness runs tasks in a deterministic text simulator with state transitions and fault injection. Graders inspect the final world state and the complete action trajectory. The project also explores small-model training and compound execution.
Evidence & scope
Experiment reports and a reproduction guide document the setup and its limits. The development suite is small and shares instruction templates with training; its results describe that environment and do not establish production performance.