Why does compressing vision-language models hide grounding failures that accuracy scores miss?

What GHOST-Q measures

Post-training quantization of vision-language models is typically assessed through aggregate task accuracy and memory savings. Preserving a headline score does not guarantee preservation of visual grounding behavior.

GHOST-Q is a cross-precision controlled evaluation of three 8B VLM families under FP16, INT8, and NF4 across utility and hallucination-sensitive benchmarks. Rather than comparing only aggregate accuracy, the authors pair FP16 and quantized predictions item by-item to quantify how compression redistributes grounding successes and failures.

The method focuses on redistribution. Each test item receives two predictions. Differences are tallied per condition. This pairing reveals shifts that average scores erase.

Accuracy stays stable while grounding breaks

Five of six quantized variants preserve MMStar accuracy within ±2 percentage points. That sounds reassuring. Yet 10 of 36 paired effects remain significant after false-discovery-rate correction. Nine of those significant effects appear on hallucination-sensitive conditions.

The mismatch means a model can pass the standard leaderboard while quietly changing what it claims to see. The paper does not say whether any variant crosses the ±2 threshold. It does not describe the nature of the grounding failures beyond their statistical significance.

The false-discovery-rate correction matters. Without it, more paired effects would appear significant. The authors chose a conservative threshold. They do not report the uncorrected counts. They do not break down which architectures contribute the nine hallucination-sensitive effects. Severity varies by architecture and precision, but the text does not quantify that variation.

Memory drops but latency does not always follow

Same-device A100 profiling demonstrates that substantial memory reduction does not necessarily mean lower inference latency. The phrase "substantial memory reduction" appears without a number. The paper does not give latency figures for any precision. It does not say whether latency sometimes increases. It does not specify batch sizes or sequence lengths used in the profiling.

The finding warns against assuming a direct trade-off between footprint and speed. Profiling runs on a single GPU type. Results on other hardware are not mentioned. The authors do not discuss power draw or throughput. They do not compare against quantization-aware training.

Generation behavior changes in ways benchmarks ignore

An open-ended AMBER audit reveals strong generation budget censoring whose severity varies by architecture and precision. The audit is described as open-ended. The term "generation budget censoring" is not defined in the abstract. The paper does not say what AMBER measures beyond this phrase. It does not report which architectures suffer most. It does not give a numeric range for the severity.

The authors conclude that quantized VLMs should be evaluated jointly for aggregate utility, grounding reliability, generation behavior, and realized deployment efficiency. That list contains four criteria. The abstract does not propose a unified metric. It does not say whether any existing benchmark covers all four. It does not mention plans to release GHOST-Q code or data. The submission targets IEEE ICASSP 2027. The paper is five pages long.