社内勉強会資料_Skywork-MoE .

Skywork-MoE と呼ばれる、MoE言語モデルのトレーニング手法について説明しています。

This presentation explains the training technique for the MoE language model, "Skywork-MoE".
Skywork Moe is a 146B-parameter mixture-of-experts LLM that achieves competitive performance using only 22B active parameters through novel initialization and training techniques.

1. Skywork-MoE: A Deep Dive into Training
Techniques for Mixture-of-Experts
Language Models

2. Introduction
●Paper: arxiv.org/pdf/2406.06563
●Github: SkyworkAI/Skywork-MoE
●Skywork-MoE
○Mixture-of-experts LLM
○146B / 22B activated
○16 experts
○Competitive against ~70B dense models
●Investigates initialization method
●Investigates training techniques

3. Quick recap on mixture-of-experts
Hypothesis: the bigger the models, the better
Problem: bigger models are more expensive to train
Solution: sparse models which give us the same power as bigger models with
less computations, mixture-of-experts are a type of sparse model
Popular implementation in Transformer-based models: sparsify the FF layers

More info.

4. Preliminaries
●Architecture based on Switch Transformers

5. Upcycling vs. from scratch
●Learning rate scheduling critical
●Diversification of the experts essential for performance
●If the budget for MoE is very small, upcycle, otherwise train from scratch
●Rule of thumb, more experimentation needed!

6. Training techniques: Gating Logit Normalization

●Routers tend to not discriminate between experts
●Makes the router probabilities sharper
●λ = 1 seems to be the best value

7. Training techniques: Adaptive Auxiliary Loss Coefficients
●Layer-wise auxiliary loss



●Increase the auxiliary loss coefficients when the token dropping rate
increases, decrease when it decreases

8. Skywork-MoE
1st, 2nd, 3rd
Absolute performance is good but not exceptional.
Beats Mixtral 8*22B and DBRX-Instruct, the most similar models.
Competitive with ~70B models for only 22B activated.

9. Additional experiments

10. Discussion
●Why use token dropping?
○Token dropping happens when experts receive more tokens than they should
○Artifact of the compute architecture which required all experts to receive the same number of
tokens
○MegaBlocks / Megatron can compute efficiently dynamic number of tokens per expert
○Used as a regularization?



●Multimodality not experimented
MegaBlocks