Why Do Sparse Activations Still Leak Private Data in Collaborative Inference?

Collaborative inference splits a neural network between a phone and a server. The phone sends an intermediate activation instead of raw data, but a new study shows that sparsifying this activation hardly reduces the privacy risk.

What collaborative inference actually sends

A phone runs the first layers of a model. It produces an intermediate activation — a tensor of numbers — and transmits that tensor to a server. The server runs the remaining layers and returns the result. This design saves bandwidth compared to sending the full input. It also keeps the raw image or audio on the device. Two problems remain. The tensor can still be large. An eavesdropper or a curious server might reconstruct the original input from it.

The sparsity assumption

Recent work compresses the tensor by keeping only the largest values. The rest become zeros. The non-zero values and their positions are then entropy-coded. Intuition suggests this helps privacy. Fewer numbers travel across the network. Less information should leak. The paper tests this intuition directly.

How the study separates values from positions

The authors decompose a sparse activation into two parts. One part contains the retained values. The other part contains only the positions where those values sit. They then reconstruct inputs from each part in isolation. This isolates the contribution of positions from the contribution of magnitudes. The experiments cover natural-image datasets and face datasets.

Positions carry the risk

The retained values leak far less than the dense tensor. The positions leak almost as much as the dense tensor. High-fidelity reconstructions emerge from positions alone. On face data, re-identification of individuals succeeds using only the position map. The leakage persists even when the transmission cost drops to a fraction of the original. It persists when task accuracy also drops. Prior analyses treated positions as side information needed only for decoding. The study shows positions are the main carrier of private content.

What this changes for system design

Any deployment that transmits sparse activations must treat the position map as sensitive data. Encryption or secure aggregation must cover positions, not just values. Audits that measure leakage from values alone will miss the dominant risk. The code accompanying the paper lets engineers reproduce the reconstruction attacks on their own models.

Where the evidence stops

The paper does not quantify how many devices currently use sparse collaborative inference in production. It does not propose a specific mitigation beyond the audit recommendation. It does not measure leakage under differential privacy or secure multiparty computation. Those questions remain open.