Log Parsing Removes Critical Clues for Distributed GPU Failures
ORIGINAL / When Log Parsing Removes What Matters: Detecting Failures in Distributed GPU Workloads
This study introduces GF-Bench, a framework for generating diverse failures in distributed GPU workloads, and evaluates sequence-based models and LLMs on parsed versus raw logs. Its novelty lies in exposing that generic log parsing reduces cost but discards distributed execution context, undermining failure detection reliability; it is relevant for cost-sensitive analysis.
01 ABSTRACT
The authors build GF-Bench with 220 labeled two-GPU executions covering normal runs, CUDA OOM, and persistent straggler failures. On parsed logs, DeepLog, LogAnomaly, LogBERT, and LogPrompt do not reliably detect both failure classes, with the highest F1 at 0.667; LogPrompt on raw logs reaches 0.983 F1 but uses 9.0x input tokens and 8.7x per-run cost. The authors argue that generic log parsing creates a substantial trade-off between effectiveness and efficiency in distributed GPU failure detection.
02 KEY FINDINGS
- GF-Bench provides 220 labeled two-GPU executions covering normal runs, CUDA OOM, and persistent straggler failures.
- On parsed logs, the four evaluated methods cannot reliably detect both failure classes, with the highest overall F1 of 0.667.
- LLM on raw logs reaches 0.983 F1 but uses 9.0x input tokens and 8.7x per-run cost.
AI GENERATED SUMMARY / DISCOVERED BY ARXIV CS.LG