Enhancing Goodput in Large-Scale LLM Training with Nonuniform Tensor Parallelism
Training large language models (LLMs) at massive scale faces infrastructure challenges due to long runtimes and thousands of GPUs. Nonuniform tensor parallelism helps mitigate slowdowns caused by device unavailability and resource fluctuations.
Training large language models (LLMs) at massive scale brings unique infrastructure challenges, especially as jobs span thousands of GPUs and run for extended periods. The longer these jobs run, the greater the likelihood of encountering unscheduled interruptions or resource fluctuations. Even infrequent device unavailability can have outsized effects on tightly interconnected clusters, resulting in slowdowns for a given training workload. Nonuniform tensor parallelism is proposed as a method to enhance goodput by adapting to these fluctuations and improving overall training efficiency in large-scale LLM deployments.