SOTA LLM Architecture

Optimal LLM Archicture

• xPOS+BCA Encoding
• SwiGLU Activation
• MQA
• Sophia Optimizer
• Flash Attention
• Early Dropout
• Blockwise Parallel Transformer (8K)

Someone else’s optimal: https://refact.ai/blog/2023/applying-recent-innovations-to-train-model/

Positional Encoding: xPOS+BCA

• NoPE
  • https://arxiv.org/pdf/2305.19466.pdf
• xPOS (+BCA)
  • https://arxiv.org/pdf/2212.10554.pdf
  • https://github.com/sunyt32/torchscale
• See also:
  • https://news.ycombinator.com/item?id=35934700
• NTK-Aware Scaled RoPE
  • https://www.reddit.com/r/LocalLLaMA/comments/14lz7j5/ntkaware_scaled_rope_allows_llama_models_to_have/
  • https://twitter.com/Yampeleg/status/1674430869828956161
• See also Positional Interpolation
  • https://arxiv.org/abs/2306.15595
  • https://kaiokendev.github.io/til#extending-context-to-8k
• Parallel Context Windows
  • https://arxiv.org/pdf/2212.10947.pdf

Multi-Query Attention

• https://arxiv.org/abs/1911.02150.pdf
• https://twitter.com/lvwerra/status/1646193143535091719
• https://arxiv.org/pdf/2211.05102.pdf
• https://arxiv.org/pdf/2305.13245.pdf
• https://paperswithcode.com/paper/multi-task-learning-with-multi-query
• https://paperswithcode.com/paper/gqa-training-generalized-multi-query

Activation Function: SwiGLU

• https://kikaben.com/swiglu-2020/
• https://medium.com/@tariqanwarph/activation-function-and-glu-variants-for-transformer-models-a4fcbe85323f
• RAF: https://aclanthology.org/2023.findings-eacl.181.pdf
• https://arxiv.org/pdf/2109.14545.pdf
• https://www.saltdatalabs.com/blog/deep-learning-101-transformer-activation-functions-explainer-relu-leaky-relu-gelu-elu-selu-softmax-and-more
• https://towardsdatascience.com/on-the-disparity-between-swish-and-gelu-1ddde902d64b
• https://chadrick-kwag.net/relu-gelu-swish-mish-activation-function-comparison/

FlashAttention

• https://pytorch.org/blog/out-of-the-box-acceleration/
• https://arxiv.org/pdf/2205.14135.pdf
• https://github.com/HazyResearch/flash-attention
• https://www.adept.ai/blog/flashier-attention
• https://arxiv.org/abs//2306.01160
• https://ahmdtaha.medium.com/flashattention-fast-and-memory-efficient-exact-attention-with-io-awareness-2a0aec52ed3d
• https://crfm.stanford.edu/2023/01/13/flashattention.html

PagedAttention

• https://vllm.ai/

Sophia Optimizer

• https://arxiv.org/pdf/2305.14342.pdf
• https://www.marktechpost.com/2023/05/26/stanford-researchers-introduce-sophia-a-scalable-second-order-optimizer-for-language-model-pre-training/
• https://github.com/sleepingcat4/Sophia
• https://www.reddit.com/r/MachineLearning/comments/1138jpp/d_lion_an_optimizer_that_outperforms_adam/

Tokenizer

• https://www.reddit.com/r/MachineLearning/comments/13gdfw0/p_new_tokenization_method_improves_llm/
• vs SentencePiece

Training

DeepSpeed ZeRO https://www.deepspeed.ai/tutorials/zero/ https://deepspeed.readthedocs.io/en/latest/zero3.html https://www.deepspeed.ai/2021/03/07/zero3-offload.html https://lightning.ai/docs/pytorch/stable/advanced/model_parallel.html

https://github.com/EleutherAI/gpt-neox/

Early Dropout

• https://arxiv.org/abs/2303.01500

Blockwise Parallel Transformer

• https://arxiv.org/pdf/2305.19370.pdf
• https://github.com/lhao499/blockwise-parallel-transformer
• https://github.com/kyegomez/Blockwise-Parallel-Transformer
• https://www.reddit.com/r/MachineLearning/comments/13xyvgt/r_blockwise_parallel_transformer_for_long_context/
• https://github.com/kyegomez/Blockwise-Parallel-Transformer

MOE

• https://arxiv.org/abs/2204.09179

Sparsity https://arxiv.org/pdf/2111.12763.pdf https://arxiv.org/pdf/2301.00774.pdf https://neuralmagic.com/blog/sparsegpt-remove-100-billion-parameters-for-free/

Efficiency https://scale.com/blog/pytorch-improvements https://github.com/lucidrains/x-transformers/issues/29

Future

MegaByte https://arxiv.org/pdf/2305.07185.pdf

Hyena https://arxiv.org/pdf/2302.10866.pdf https://hazyresearch.stanford.edu/blog/2023-03-07-hyena

Fine Tuning

DPO https://arxiv.org/abs/2305.18290

Other Models

https://huggingface.co/blog/starcoder https://www.mosaicml.com/blog/mpt-30b https://huggingface.co/replit/replit-code-v1-3b

Inference Tricks

https://threadreaderapp.com/thread/1675023699802456066.html