Cooperative optimization method for inference on multi-chiplet large-model accelerators

In large language model inference accelerators integrating multiple compute and memory chiplets via 2.5D packaging, cross-chiplet communication during the decoding stage is bursty and highly unbalanced, causing traffic to concentrate on a small number of links and form hotspot queuing, which tends to make the network-on-package a performance bottleneck. To mitigate this bottleneck, T²-CHIP was proposed, a collaborative optimization method that characterized the distribution of decoding-stage cross-chiplet traffic over the interconnect, identified hotspot links, reallocated bandwidth resources, and adjusted task mapping to reduce hotspot cross-chiplet interactions, thereby effectively relieving decoding-stage communication congestion. Cycle-accurate network simulations show that the proposed method improves decoding-stage tail performance and overall throughput while reducing dynamic power consumption and maintaining low implementation overhead.