Benchmarking Attention for Tabular Foundation Models Across Hardware and Shapes

01 ABSTRACT

The paper notes that tabular in-context learners rely on alternating row and column attention, where row attention involves long sequences and column attention short ones, with strided memory layouts and small hidden dimensions, while efficient attention research has focused on one-dimensional sequences. The authors benchmark multiple attention backends on A100, H100, and B200 GPUs and find that the optimal backend differs between column and row attention and varies with hardware and model specifics; FlashAttention implementations tailored to each GPU generation perform overall best, but are sometimes outperformed by cuDNN for column attention at longer sequences with cross-over points depending on head dimension. The authors state that this reproducible benchmark lays the foundation for future improvements to table-native attention (author viewpoint).

02 KEY FINDINGS

1. Tabular attention differs markedly from one-dimensional language attention in sequence lengths and memory layout, but efficient attention benchmarks for the two-dimensional tabular setting were previously lacking.
2. Forward and backward throughput are measured on A100/H100/B200 across Torch SDPA, FlashAttention-2/3/4, vLLM, and SageAttention.
3. The optimal backend depends on attention type (column/row), sequence length, head dimension, and hardware; FlashAttention is overall best, cuDNN sometimes wins for long-sequence column attention, and SageAttention performs well for row attention beyond 16k rows.