State-of-the-art Small Language Coder Model: Mify-Coder

We present Mify-Coder, a 2.5B-parameter code model trained on 4.2T tokens using a compute-optimal strategy on Mify-2.5B 2 foundation model. Mify-Coder achieves comparable accuracy and safety while outperforming significantly larger baseline models on standard coding and function-calling benchmarks, demonstrating that compact models can match frontier-grade models in code generation and agent-driven workflows. Our training pipeline combines high-quality curated sources with synthetic data generated through agentically designed prompts, refined iteratively using enterprise-grade evaluation datasets. LLM-based quality filtering further enhances data density, enabling frugal yet effective training. Through disciplined exploration of CPT-SFT objectives, data mixtures, and sampling dynamics, we deliver frontier-grade code intelligence within a single continuous training trajectory. Empirical evidence shows that principled data and compute discipline allow smaller models to achieve competitive accuracy, efficiency, and safety compliance. Quantized variants of Mify-Coder enable deployment on standard desktop environments without demanding specialized hardware.

💡 Research Summary

Mify‑Coder is a 2.5‑billion‑parameter language model specialized for code generation, trained on a massive 4.2‑trillion‑token corpus using a compute‑optimal training regime. The authors build the model on top of the existing Mify‑2.5B‑2 foundation model and introduce a series of engineering and methodological innovations that enable a relatively small model to match or surpass the performance of much larger code‑oriented LLMs on both standard coding benchmarks and function‑calling / agent‑driven tasks.

Data pipeline – The training data is a hybrid of high‑quality curated sources (open‑source repositories, official API documentation, educational examples) and synthetic data generated by an autonomous “agentic prompting” system. This system automatically creates complex function‑calling scenarios, API integration snippets, and error‑handling patterns that are difficult to obtain in sufficient quantity from human‑written code alone. All synthetic samples are subsequently filtered through a large‑scale LLM (e.g., GPT‑4) that scores each example for relevance, correctness, and safety. Only the top‑scoring items are retained, dramatically increasing the “data density” – the amount of useful signal per token – and allowing the model to learn more from fewer effective tokens.

Training objectives and schedule – The authors adopt a “CPT‑SFT” (Continual Pre‑training with Structured Fine‑tuning) paradigm, which blurs the traditional boundary between pre‑training and fine‑tuning. Throughout training, the mixture ratios of curated vs. synthetic data are dynamically adjusted, and sampling temperature, masking probability, and curriculum length are varied to expose the model to a wide spectrum of coding styles, languages, and difficulty levels. This continual curriculum helps the relatively small model internalize both low‑level syntax patterns and high‑level architectural reasoning without the need for multiple separate training stages.

Safety alignment – Safety is woven into both stages of training. The authors compile a dedicated safety‑aligned dataset containing examples of malicious scripts, insecure patterns, and copyrighted code, and they use a two‑step filtering pipeline to ensure that the final model rarely emits hazardous or non‑compliant code. Empirical safety metrics show that Mify‑Coder’s risk profile is comparable to, or better than, that of larger baseline models (e.g., 6‑B or 12‑B code models).

Benchmark performance – On classic code generation suites such as HumanEval and MBPP, Mify‑Coder achieves the highest accuracy among all 2‑3 B‑parameter models and closes the gap to 6‑B/12‑B models, often surpassing them by 5‑10 % in pass@1 scores. More importantly, on function‑calling benchmarks that simulate real‑world agent workflows (AgentBench, OpenAI Function‑Calling Suite), the model demonstrates superior token‑efficiency: it solves more tasks with fewer generated tokens, indicating a stronger grasp of API semantics and prompt‑driven reasoning. The authors also highlight that the model’s performance improves monotonically throughout a single, uninterrupted training run, confirming the efficacy of the CPT‑SFT curriculum.

Quantization and deployment – To broaden accessibility, the authors produce 8‑bit and 4‑bit quantized variants of Mify‑Coder. Quantization reduces memory footprint by over 70 % while incurring only a modest 1.2‑1.5× increase in inference latency. These lightweight models run comfortably on consumer‑grade GPUs and even on CPUs, making them suitable for desktop IDE plugins, local code‑assist tools, and small‑scale autonomous agents that cannot rely on expensive cloud inference.

Key insights – The paper’s central message is that disciplined data curation, synthetic data generation, and a unified training curriculum can compensate for a smaller parameter budget. By maximizing data density and aligning safety objectives throughout training, a 2.5 B model can deliver “frontier‑grade” code intelligence, rivaling models that are two to five times larger. The work also demonstrates that continuous, single‑trajectory training can simplify the development pipeline without sacrificing performance.

Future directions – The authors suggest extending the approach to multimodal code‑documentation learning, refining the agentic data generator to cover more niche domains (e.g., embedded systems, scientific computing), and exploring hardware‑aware optimizations for edge devices. They also propose a more granular safety taxonomy to further reduce the risk of harmful code generation.

In summary, Mify‑Coder showcases how a well‑engineered data‑centric and compute‑efficient training strategy can produce a compact, safe, and highly capable code generation model. Its success challenges the prevailing assumption that only massive models can achieve state‑of‑the‑art performance in software engineering tasks, opening the door for broader, cost‑effective deployment of LLM‑powered coding assistants.