Can Hybrid-Order Training Let Memory-Poor Devices Join Federated Learning?

The Memory Wall That Stops Edge Devices

Federated learning promises to train models on data that never leaves the device. Phones, sensors, and microcontrollers all hold pieces of the puzzle. The central server stitches their updates into a global model. But there is a catch. Standard training uses backpropagation, a first-order method that needs to store intermediate activations for every layer. On a tiny microcontroller that memory simply does not exist.

The alternative, zeroth-order optimization, estimates gradients by probing the loss function. It needs almost no memory. It also converges so slowly that the model may never reach useful accuracy. The paper frames this as a dilemma: either the device has enough RAM for backpropagation, or it drops out of the training round. Memory is the gatekeeper.

How HO-FL Splits the Work

HO-FL cuts the model at a configurable boundary. Layers below the boundary — the bottom segment — are trained with zeroth-order updates. Layers above — the top segment — use first-order backpropagation. Each device chooses its own boundary based on how much memory it can spare. A phone might run backpropagation through most of the network. A sensor might only afford it for the final classifier head. Both devices still contribute to the same global model. The server averages their updates without knowing which layers used which optimizer. The paper calls this a hybrid-order framework. The order boundary is a local decision. No coordination round is needed to align the split points. Code is released at a URL the abstract does not spell out.

The Trade-Off That Emerges From Uneven Splits

The convergence analysis uncovers a structural tension. Clients that allocate more layers to first-order training produce lower-variance updates. Their gradients are exact rather than estimated. If the server samples those clients more often, the global model improves faster. But the data held by memory-poor clients then gets underrepresented. The paper links this to the bias and variance of actual multi-step local updates. It formulates a sampling optimization problem that balances update quality against data coverage. A practical dimension-aware approximation lets the server implement the balance with direct model averaging. The approximation uses the size of each client's first-order segment as a proxy for update reliability. How many clients fall into each memory tier in real deployments is not reported.

What the Language Experiments Show

Experiments run on language tasks. They measure task performance, client memory consumption, and sampling behavior under data heterogeneity. The results show that hybrid-order local training retains much of the full first-order performance while cutting client memory requirements substantially. The paper does not quote a single percentage or absolute memory figure in the abstract. It states 30 pages and 2 figures in the manuscript. The submission date is 30 September 2026. The authors are Qiyuan Chen, Xian Wu, Yanan Ma, and Xianhao Chen. The DOI is 10.48550/arXiv.2609.39074.