Foundation Models

Foundation models are characterized by their scale (billions of parameters), the breadth of their training data (diverse internet-scale text, code, and increasingly images and audio), and their emergent capabilities—behaviors that appear at scale and were not explicitly trained for, such as in-context learning and multi-step reasoning.

• • Scale: billions of parameters trained on trillions of tokens.
• • Generality: capable of many tasks with minimal task-specific data.
• • Emergence: capabilities arise at scale that were not present in smaller models.

Organizations adapt foundation models along a spectrum. At one end, prompt engineering requires no changes to the model. Moving further, retrieval-augmented generation (RAG) grounds the model in proprietary data. Fine-tuning modifies model weights for specific behaviors. At the other end, pre-training from scratch (rare and very expensive) gives full control.

Closed models (GPT-4, Claude) are accessed via API. Open-weight models (Llama 3, Mistral, Gemma) release model weights publicly, enabling on-premise deployment and full customization. The choice involves trade-offs around capability, cost, privacy, and control.