AgriGPT-Omni: A Unified Speech-Vision-Text Framework for Multilingual Agricultural Intelligence

Despite rapid advances in multimodal large language models, agricultural applications remain constrained by the lack of multilingual speech data, unified multimodal architectures, and comprehensive evaluation benchmarks. To address these challenges, we present AgriGPT-Omni, an agricultural omni-framework that integrates speech, vision, and text in a unified framework. First, we construct a scalable data synthesis and collection pipeline that converts agricultural texts and images into training data, resulting in the largest agricultural speech dataset to date, including 492K synthetic and 1.4K real speech samples across six languages. Second, based on this, we train the first agricultural omni-model via a three-stage paradigm: textual knowledge injection, progressive multimodal alignment, and GRPO-based reinforcement learning, enabling unified reasoning across languages and modalities. Third, we propose AgriBench-Omni-2K, the first tri-modal benchmark for agriculture, covering diverse speech-vision-text tasks and multilingual slices, with standardized protocols and reproducible tools. Experiments show that AgriGPT-Omni significantly outperforms general-purpose baselines on multilingual and multimodal reasoning as well as real-world speech understanding. All models, data, benchmarks, and code will be released to promote reproducible research, inclusive agricultural intelligence, and sustainable AI development for low-resource regions.

💡 Research Summary

This paper introduces AgriGPT-Omni, the first unified speech-vision-text omni-modal framework specifically designed for agricultural intelligence. It addresses critical gaps in current agricultural AI: the scarcity of multilingual speech data, the lack of a unified architecture for all three modalities, and the absence of comprehensive evaluation benchmarks.

The work is built upon three foundational pillars. First, for data, the authors construct the largest agricultural speech dataset to date. They leverage a scalable hybrid pipeline, translating existing agricultural text (Agri-342K) and image-text pairs (AgriVL-150K) into six languages (Chinese, Sichuan dialect, Cantonese, English, Japanese, Korean) and synthesizing them into approximately 492,000 high-quality speech samples using the CosyVoice2 TTS system. This is complemented by 1,400 real human speech recordings to enhance robustness to real-world accents and noise.

Second, for the model, AgriGPT-Omni is built upon Qwen-2.5-Omni and trained via a meticulous three-stage curriculum. Stage 1: Textual Knowledge Injection involves continued pre-training on agricultural corpora (2.2B tokens) and supervised fine-tuning on text-only and speech-text QA pairs to establish domain expertise and basic speech comprehension. Stage 2: Progressive Multimodal Alignment first aligns the vision and speech encoders to the language space by training only the modality-specific adapters while keeping the encoders and language backbone frozen. Subsequently, the language backbone is unfrozen for joint instruction tuning on large-scale Vision QA, high-quality GPT-4o refined samples, and tri-modal (speech+image) QA data, solidifying unified cross-modal reasoning. Stage 3: Preference Optimization employs Group Relative Policy Optimization (GRPO) on speech multiple-choice and transcription tasks to refine answer accuracy and output stability.

Third, for evaluation, the paper proposes AgriBench-Omni-2K, the first tri-modal benchmark for agriculture. It encompasses four task types—Audio QA, Audio+Text Multiple Choice, Multimodal (Audio+Image) QA, and Multimodal (Audio+Image+Text) Multiple Choice—across the same six languages, totaling 1,500 samples. An additional set of 586 real human speech recordings is used for evaluating real-world robustness. Rigorous de-duplication ensures no overlap with training data.

Experimental results demonstrate that AgriGPT-Omni significantly outperforms state-of-the-art general-purpose multimodal models (e.g., InternVL, LLaVA, Qwen-VL, Gemini) on established agricultural text and vision-language benchmarks (AgriBench-13K, AgriBench-VL-4K) across standard metrics like BLEU, METEOR, and ROUGE. More importantly, it achieves superior performance on the novel and challenging tasks within the AgriBench-Omni-2K benchmark, proving its effectiveness in unified multilingual and multimodal reasoning. An ablation study validates the contribution of each training stage, and an analysis comparing performance on synthetic versus real speech confirms the model’s practical utility.

In conclusion, AgriGPT-Omni represents a significant leap from “text-image understanding” towards true “omni-modal interaction” in agriculture. By releasing all models, datasets, benchmarks, and code, the authors aim to foster reproducible research and promote inclusive, sustainable AI development for low-resource agricultural regions worldwide.