Progressive LLM Training

Progressive training for LLMs with 8-GPU support for 松尾研LLMコンペ2025.

Quick Start

# Setup project
git clone <repository-url>
cd progressive-llm-training

# Install dependencies
pip install -r requirements.txt

# Start training
python scripts/train_progressive.py --config config/training_config_gemma3_1b.yaml
./scripts/train_gemma3_1b_8gpu.sh --strategy ddp

Training Stages

1. basic_cot - Basic reasoning
2. math_reasoning - Math with OpenR1-Math-220k
3. complex_reasoning - Complex reasoning with Mixture-of-Thoughts

Commands

pip install -r requirements.txt                                                     # Install dependencies
python scripts/train_progressive.py --config config/training_config_gemma3_1b.yaml # Single GPU
./scripts/train_gemma3_1b_8gpu.sh --strategy ddp                                   # 8 GPUs (DDP)
python scripts/train_ddp_simple.py config/training_config_gemma3_1b_8gpu_ddp.yaml  # 8 GPUs (Simple)
pytest                                                                              # Run tests

Key Files

• config/training_config_gemma3_1b_8gpu_deepspeed.yaml - 8-GPU config
• scripts/train_progressive.py - Main training script
• scripts/train_gemma3_1b_8gpu.sh - 8-GPU launcher
• src/progressive_model.py - Core model implementation

Ready to train! 🚀