Vision language and embedding models

Two model types look different from a chat LLM and turn out to be close cousins. A vision language model (VLM) runs a vision encoder over an image and turns it into a few hundred to a few thousand visual tokens. Those tokens sit in the prompt next to the text, and from there the language model treats them like any other input. So an image costs you exactly what extra prompt tokens cost: a longer prefill, more KV cache, and a later first token.

An embedding model is the other half of the story. It reads a piece of text in one forward pass and returns a vector. It never decodes, so there is no inter-token latency to worry about. What matters is how many tokens per second you can push through, which comes down to batching and how fast you can tokenize.