What happens when sparsity, quantization, and low-rank approximations work together instead of alone?

The wall each method hits alone

Large language models cost too much to run. Compute and memory demands grow faster than budgets allow. Engineers have three main tools: sparsity to cut operations, quantization to shrink memory traffic, and low-rank approximations to keep accuracy. The thesis states each tool hits an accuracy-efficiency wall when used by itself. Sparsity alone leaves gaps in representation. Quantization alone adds noise that accumulates. Low-rank alone cannot capture all directions. The document argues the wall appears because the methods fight for the same degrees of freedom.

Five components that make the Trinity work

The thesis builds five systems. MKOR approximates curvature with block-diagonal sparsity and low-rank inversion. It keeps numerical stability for quantized optimizer states. Curvature update complexity drops from $O(d^3)$ to $O(d^2)$. Convergence accelerates by up to 1.85x over KFAC. SLoPe speeds training by up to 1.25x through a double-pruned backward pass for N:M sparsity. Low-rank lazy adapters activate in the final 1% of training to recover accuracy. OPTIMA stabilizes static masks without training. It formulates weight reconstruction as globally optimal column-wise quadratic programs. Zero-shot accuracy improves by up to 3.97%. PATCH learns a dynamic hybrid sparsity ratio between 0% and 50% given a fine-tuning budget. It breaks the ceiling of static masks and yields up to 1.38x speedups. SLiM realizes the full Trinity in one shot. Mathematically derived low-rank adapters recover information lost to quantization and sparsity. Accuracy improves by up to 5.66% over state-of-the-art methods. At equal parameter budgets SLiM outperforms uncompressed dense models by 0.6%.

What the joint approach changes for training and inference

Joint application changes the trade-off curve, the thesis shows. MKOR and SLoPe act during pretraining. They modify optimizer updates and backward passes. OPTIMA and PATCH act after training. They compress weights with or without fine-tuning. SLiM combines everything in a single pass. The document reports numbers for each component separately. It does not report a single end-to-end run where all five run together on one model. The thesis spans 156 pages. Chapters extend five prior papers: MKOR (arXiv:2306.01685), SLoPe (arXiv:2405.16325), OPTIMA (arXiv:2512.13886), PATCH (arXiv:2509.23410), and SLiM (arXiv:2410.09615). Submission date is 25 Aug 2026.

Where the thesis leaves open questions

The abstract does not specify model sizes used for the reported speedups. It does not name hardware platforms. It does not give memory footprint reductions in gigabytes. It does not discuss inference latency at batch sizes typical for serving. The thesis is a PhD dissertation. Peer-reviewed publications of the individual chapters exist on arXiv. The unified framework itself has not appeared in a conference or journal venue as of the submission date.