Can a local language model turn a bug report into a crashing input?

What the experiment actually measured

Vulnerability reproduction means taking a textual description of a flaw and producing an input that crashes the vulnerable build. The task sits closer to test generation than to code comprehension because the agent must execute the program rather than only read it. The authors chose this target because local open-weight models can run on a single consumer GPU without sending code to a hosted service.

They evaluated twelve configurations spanning three model generations, two architectures, and three quantization levels. Each configuration ran three times on each of ten CyberGym tasks under one harness. The full artefact was released alongside the preprint.

Model size decides the ceiling

Every configuration, down to the 4B model, solved most of the six easier tasks. Even 4B worked. Configurations separated only on the remaining four harder tasks. For those harder tasks the 27B model was required. Smaller models did not reach the same pass rate no matter how many runs they received.

Quantization did not change what the models solved in any matched pair. It changed how fast they got there. The mixture-of-experts architecture reached the same scores in less wall clock time than dense counterparts of comparable size.

What the agent trajectories reveal

Almost every run examined, including those that fail, names the vulnerable function and states the correct hypothesis. The difference between a solved task and an unsolved one is whether that hypothesis becomes a working input. Failures tend to stall at the translation from hypothesis to concrete exploit rather than at the analysis stage.

The paper spans thirty-five pages with six figures and five tables. It is a preprint of a manuscript under review at the Journal of Systems and Software.