The LLM Scientist Roadmap¶
约 83 个字 预计阅读时间不到 1 分钟
1. The LLM architecture¶
- Architectural Overview
- Tokenization
- Attention mechanisms
- Sampling techniques
2. Pre-training models¶
- Data preparation
- Distributed training
- Training optimization
- Monitoring
- Storage & chat templates
- Synthetic data generation
- Data enhancement
- Quality filtering
3. Post-training datasets¶
- Training techniques
- Training parameters
- Distributed training
- Monitoring
4. Supervised Fine-Tuning¶
- Rejection sampling
- Direct Preference Optimization
- Reward model
- Reinforcement Learning
5. Preference alignment¶
- Automated benchmarks
- Human evaluation
- Model-based evaluation
- Feedback signal
6. Evaluation¶
- Base techniques
- GGUP and llama.cpp
- GPTQ & AWQ
- SmoothQuant & ZeroQuant
7. Quantization¶
- Model merging
- Multimodal models
- Interpretability
- Test-time compute