MegaGraph accepted by ICCD 2026: automated hybrid parallelism for large-scale Graph Transformer training

👏 Paper title: MegaGraph: Towards Efficient Training of Large-scale Graph Transformers with Automated Hybrid Parallelism.

MegaGraph enables efficient Graph Transformer training on large-scale graphs through automated hybrid parallelism. Graph Transformers offer powerful representation capabilities, but their attention and topology-aware bias matrices create substantial memory overhead, while graph embedding and data construction cause severe workload imbalance.

The framework combines graph-aware context parallelism, heterogeneous pipeline parallelism, and hybrid data parallelism. These strategies distribute attention computation and topology-aware biases across GPUs, balance pipeline workloads through non-uniform layer allocation, and scale throughput along the graph-data dimension.

MegaGraph also provides an automatic search engine that follows a Profile → Model → Search workflow. Using lightweight profiling, cost models, and memory-aware pruning, it identifies an efficient configuration from the large hybrid-parallelism search space within seconds.

By adapting parallel execution and configuration search to graph-specific workloads, MegaGraph provides a practical path for scaling Graph Transformer training without changing the model architecture or training objective.