dictionary

Inference

While training is the resource-intensive process of teaching a model, inference is the act of actually using the model. When you type a prompt into ChatGPT and it generates a response, that generation process is a series of inference steps. Optimizing inference is critical for reducing latency and cloud costs in production systems.