MoEDCoder | Diffusion Mixture-of-Experts SLM
Sobhan Bahrami
MoEDCoder: Mixture-of-Experts Masked Diffusion Language Model
Trained a novel Diffusion Small Specialized Language Model (SLM) with Mixture of Experts (10 Experts in total) from scratch, accompanied by a custom evaluation harness and dynamic router stabilization framework.
Key Architectural Innovations:
Diffusion-Based Code Synthesis: Implemented discrete masked-diffusion language modeling for structured code generation.
LR-MoE Framework-Supervised Routing: Overcame early-phase routing collapse in diffusion steps, achieving a 20.5% perplexity improvement (85.72 -> 68.11).
Inference Router Caching: Established routing trajectory stability (>0.98), enabling caching mechanisms that significantly reduce computation.
Distributed Training: Orchestrated multi-GPU training passes using PyTorch, Ray, and mixed-precision optimization.
Research & Verification:
Target track: NeurIPS 2026 ENLSP-V.
Keynote & Speaker proposal: PyCon Armenia 2026.
Like this project
