Parallel Training of Large Language Models

课程大纲

• 大模型并行训练 [课件]

推荐阅读材料

• [论文] Fast and Memory-Efficient Exact Attention with IO-Awareness
• [论文]Ring Attention with Blockwise Transformers for Near-Infinite Context