Residual-Aware Neural Spectral Capacity for Heterogeneous Resource Allocation in Transformers
ORIGINAL / Neural Spectral Capacity with Residual Context for Heterogeneous Resource Allocation within Transformers
Introduces Residual-Aware Neural Spectral Capacity (ResNSC), which improves Transformer performance prediction by incorporating residual context and enhances the allocator to jointly allocate residual and feed-forward widths, validated in pretraining and pruning tasks.
01 ABSTRACT
Neural Spectral Capacity (NSC) is a heuristic for predicting relative trained performance from architectural specifications, but it scores projections independently and assigns identical scores to compatible reorderings. This work proposes ResNSC, which uses a specification-derived statistical reference for accumulated residual updates to determine input and residual-state statistics in each projection's capacity calculation, and augments the NSC-DP allocator with residual history. Experiments show that at 121M and 374M pretraining scales, selected architectures achieve lower perplexity than uniform, variable-width, and tapered allocations at comparable parameter budgets, and in LLaMA-7B pruning, ResNSC-DP improves eight-task average accuracy by 3.28–5.46 percentage points.
02 KEY FINDINGS
- ResNSC introduces a statistical reference for residual updates, addressing NSC's independent scoring of projections.
- ResNSC-DP jointly allocates residual and feed-forward widths and provides certified bounds on the best score within a finite candidate set.
- ResNSC-DP outperforms baseline allocation strategies in pretraining and pruning, and paired reversal experiments reveal performance differences hidden by position-independent scoring.
AI GENERATED SUMMARY / DISCOVERED BY ARXIV CS.LG