Abstract

Sparse Mixture of Experts (SMoE) architectures reduce computational cost by activating only a subset of experts per token, yet they often retain large memory footprints and exhibit significant redundancy, both within and across layers. We propose GMoE, a sparse MoE architecture designed to explicitly address these inefficiencies. Instead of maintaining separate expert sets for each layer, GMoE uses Global Experts shared across all layers and adds a Local Expert per layer for layer-specific adaptation. This architecture reuses Global Experts across layers, thereby mitigating inter-layer redundancy while substantially reducing model parameters. In addition, we introduce a Global Router with a GRU-based recurrent component shared across layers and layer-specific routing heads that propagate routing logits across layers. This routing mechanism couples routing decisions across layers, progressively refines routing paths, and helps mitigate intra-layer redundancy. Across diverse language modeling corpora and downstream benchmarks, GMoE remains competitive while using substantially fewer parameters. Routing path analyzes and an ablation study show that GMoE reduces cross-layer routing concentration and increases path diversity, with the Global Experts, the Local Expert, and the Global Router all contributing to the gains.

Mixture-of-Experts (MoE) models improve AI efficiency by activating only a small number of specialized components, known as experts , for each input. But as these models grow deeper, similar experts are often repeated across layers, increasing model size and memory without adding much new capability.

A research team led by Professor Taehwan Kim of the Graduate School of Artificial Intelligence at UNIST has developed GMoE (Global Mixture of Experts), a new architecture that allows layers to share experts instead. The approach substantially reduces model parameters while maintaining comparable language performance.

GMoE replaces separate expert sets at each layer with Global Experts shared throughout the model, while retaining one Local Expert per layer for specialized functions. This allows useful expertise to be reused rather than repeatedly stored across the network.

The team also improved how the model selects experts. Conventional MoE models make these decisions independently at each layer, which can repeatedly direct inputs through the same expert combinations. GMoE carries information from earlier selections forward, allowing subsequent layers to make more varied choices.

Tests across seven language tasks, including reading comprehension, grammatical judgment, and commonsense reasoning, showed comparable performance with substantially fewer parameters. In a medium-scale model, GMoE reduced the parameter count by about 63%, from 549 million to 204 million, while average accuracy remained nearly unchanged at 39.51%, compared with 39.55% for the conventional model.

GMoE also produced 81,561 distinct expert paths, more than three times as many as the conventional approach. The share of inputs following the most common path fell to 11.15%, compared with 25.65% to 45.55% in the baseline models.

“Conventional MoE models can repeat similar functions across layers while relying heavily on a limited number of expert combinations,” said Professor Kim. “GMoE addresses both problems by sharing experts across layers and connecting how those experts are selected.” He added, “The results show that language models can maintain competitive performance with far fewer parameters while using a broader range of expert combinations.”

Geonwoo Hong, a master's student in the Graduate School of Artificial Intelligence at UNIST, served as first author. The study was accepted to the Annual Meeting of the Association for Computational Linguistics (ACL 2026), a leading international conference in natural language processing. The research code is publicly available through the team's project repository.

Journal Reference

Geonwoo Hong and Taehwan Kim, "GMoE: Global Mixture of Experts with Logit Propagation,"   ACL '26,  (2026).