Can compact language models answer compliance questions without making up rules?

The problem with small models and big rulebooks

Financial institutions operate under dense rulebooks that change often. Answering a compliance question correctly requires more than fluent text. It requires verifiable grounding in the authoritative text. Large language models look attractive for this task. Yet the models that firms can realistically deploy on-premise are compact ones. Compact models hallucinate obligations.

How the retriever was rebuilt

The paper studies whether a carefully domain-adapted retrieval-augmented generation pipeline closes that gap. The retriever is built in three stages on top of LegalBERT. First stage applies entailment tuning that recasts question-passage matching as premise-hypothesis reconstruction. Second stage adds contrastive tuning with in-batch negatives. Third stage fuses scores with BM25. The generator is a compact model between 2B and 12B parameters served under 4-bit quantization. It is either prompted or adapted with retrieval-aware fine-tuning through LoRA.

Benchmark results show a real retrieval jump

On ObliQA, a question-answering benchmark built from the Abu Dhabi Global Market rulebooks, the staged retriever raises Recall@10 from 0.256 to 0.774. It outperforms BM25 at 0.678 and E5-large-v2 at 0.758. E5-large-v2 is the strongest general-purpose dense encoder the authors tested.

Generation gains do not equal grounding

RAFT-LoRA then improves the composite RePASs answer-quality score for every model the authors could adapt. The largest gain appears on the weakest model. However the adapted models do not transfer to Australian case-law questions. A closed-book model that receives no passages at all scores within 0.011 RePASs of the full pipeline. That model produces answers that cite nothing and misstate obligations. The retrieval gain is therefore measured directly. The generation gain is a gain in RePASs rather than demonstrated grounding. Grounding itself requires an evaluation protocol that RePASs does not provide. The paper is currently under review.