Para-Pipe extracts 115% ML operator parallelism from SoCs

Why this matters now

Edge devices and mobile SoCs often fail to deliver expected performance when running ML inference or training. Size and power constraints limit GPU/NPU resources, and operators that seem ready to run in parallel still leave performance on the table. This is where Para-Pipe rethinks the problem from operator-level parallelism scheduling.

Background: where existing schedulers hit a wall

ML frameworks usually execute computational graphs with data parallelism or layer-wise pipeline parallelism. SoCs add further complexity with big.LITTLE CPUs, GPUs, NPUs, and DSPs. Existing schedulers struggle with operator-level parallelism because they operate at coarse granularity and cannot fully exploit fine-grained resources inside the graph.

Key mechanism: coarse/fine ILP and fan-in classification

Para-Pipe splits scheduling into two stages. Coarse ILP fixes the execution order at large subgraph boundaries, while fine ILP exploits dependencies among individual operators within that boundary for simultaneous execution. The paper further classifies structures into fan-in and linear-chain patterns, tuning parallelism differently for each to gain throughput without sacrificing latency.

These structures are evaluated on two real SoCs: A311D combines ARM big.LITTLE with a GPU, while BST pairs an NPU with two DSPs. Covering both cases matters because CPU-heavy and accelerator-heavy paths often share the same dataflow and need different operator mapping strategies.

Results: latency, throughput, and energy together

Key numbers are as follows. Against a pipe-only baseline, latency improves by up to 115%. A para-only approach sacrifices about 26% throughput. Hybrid-A reaches 7.9% throughput and 33.1% latency improvement; Hybrid-B reaches 6.2% throughput and 11.5% latency. Energy efficiency reaches 11.0% under throughput-optimal settings and 23.3% under latency-optimal settings.

The paper tests two to four models each for Inception and transformer architectures. Different operator patterns across models cause throughput variation in real deployment, showing that architecture-aware scheduling is not optional.

Why this matters

SoC scheduling has traditionally relied on frameworks handling parallelism automatically. Para-Pipe is notable because it analyzes the graph structure itself and separates coarse and fine stages, enabling simultaneous latency and throughput improvement on the same hardware. The ability to jointly optimize latency and energy is especially relevant for mobile inference services where battery life and response speed both matter.

Implications and limitations

Explicitly separating coarse and fine ILP is applicable not only to current NPUs and GPUs but also to future compilers and runtime systems. Adding transformations solely to increase fan-in can increase communication volume, so real-world deployment must weigh memory bandwidth. Because SoC memory topologies differ, applying the A311D/BST results directly to other chips requires caution. The direction of designing at operator parallelism granularity remains clear.