dictionary

Mixture of Experts (MoE)

Instead of activating the entire massive neural network for every prompt, an MoE model uses a "router" to send the prompt only to the relevant specialist sub-networks (e.g., sending a coding question to the coding expert). This design allows models to have massive parameter counts while keeping inference fast and compute costs low.