In the engine room of every AI application, algorithms are hard at work processing data using a unique language composed of tokens.
AI tokens are small units of information derived from breaking down larger data blocks. AI models utilize these tokens to learn patterns and relationships, enabling capabilities like forecasting, creation, and logical deduction. The speed at which tokens are processed directly correlates to how quickly the model can learn and react. The ultimate objective is to secure the quickest processing speeds and the lowest cost per token, thereby optimizing AI infrastructure to boost revenue potential.
AI factories — a new breed of data centers built to speed up AI workloads — effectively process vast amounts of tokens. They transform these tokens from the raw language of AI into a valuable currency: intelligence.
By leveraging AI factories, enterprises can utilize state-of-the-art full-stack computing solutions to handle more tokens at a reduced computational cost, creating added value for customers. In a recent instance, the combination of software optimizations and the latest NVIDIA GPUs slashed the cost per token by 20x compared to older, unoptimized hardware — generating 25x more revenue in just four weeks.
Through the efficient processing of tokens, AI factories are essentially manufacturing intelligence — the most critical asset in this new AI-driven industrial revolution.
How Does Tokenization Turn Data Into AI-Readable Tokens?
Whether a transformer AI model is handling text, images, audio, video, or other types of data, it first translates the information into tokens. This conversion step is called tokenization.
Efficient tokenization is key to lowering the computing power needed for training and inference. There are many tokenization techniques available. Tokenizers designed for specific data types often use a smaller vocabulary, which results in fewer tokens to process.
For large language models (LLMs), short words often correspond to a single token, while longer ones are broken into two or more.
For example, the word "darkness" is split into two tokens: "dark" and "ness," each assigned a numerical ID, such as 217 and 655. Similarly, the word "brightness" is split into "bright" and "ness," with corresponding numbers like 491 and 655.
Here, the shared numerical value for "ness" helps the AI model recognize a relationship between the two words. Conversely, a tokenizer might assign different numerical representations to the same word depending on the context.
Take the word "lie," for example. It could mean to recline or to tell a falsehood. During training, the model learns to distinguish these contexts and assigns distinct token numbers to each meaning.
For visual models dealing with images or video, tokenizers map visual elements like pixels or voxels into discrete token sequences.
Models handling audio may convert short clips into spectrograms — visual representations of sound waves over time — which are then processed as images. Other audio applications might focus on capturing the meaning of speech, using semantic tokenizers that represent language or context rather than just acoustic details.
How Are Tokens Used During AI Training?
The process of training an AI model begins with tokenizing the training dataset.
Depending on the data size, the token count can reach into the billions or even trillions. According to pretraining scaling laws, the more tokens used in training, the higher the quality of the resulting model.
During pretraining, the model is tested by being shown a sample set of tokens and asked to predict the next one. Based on the accuracy of its guess, the model updates its internal parameters to improve future predictions. This cycle repeats until the model learns from its errors and achieves a target accuracy level, a phase known as model convergence.
After pretraining, models undergo post-training to further refine their capabilities. Here, they continue learning on a subset of tokens relevant to their specific deployment. This could involve domain-specific data for fields like law, medicine, or finance, or tokens that help the model master specific tasks like reasoning, chatting, or translating. The end goal is a model that generates the correct tokens to answer a user's query accurately — a skill known as inference.
How Are Tokens Used During AI Inference and Reasoning?
During inference, an AI accepts a prompt — which could be text, an image, audio, video, sensor data, or even a gene sequence — and translates it into a series of tokens. The model processes these input tokens, formulates a response as tokens, and then translates that back into a format the user expects.
Input and output formats can differ significantly, such as in models that translate English text into Japanese images or convert text prompts into pictures.
To fully grasp a prompt, AI models need to process multiple tokens simultaneously. Many models have a defined limit called a context window, and different applications require different window sizes.
A model capable of processing a few thousand tokens at once might handle a single high-resolution photo or a few pages of text. With a context window of tens of thousands of tokens, a model could summarize an entire novel or a lengthy podcast. Some advanced models offer context windows of a million tokens or more, enabling the analysis of massive datasets in one go.
Reasoning AI models, a cutting-edge development in LLMs, handle complex queries by processing tokens in novel ways. In addition to standard input and output, these models generate numerous "reasoning tokens" over minutes or hours as they "think" through a problem.
These reasoning tokens lead to better answers on complex tasks, much like a person formulates a better solution given time to deliberate. However, this can increase the token count per prompt by over 100x compared to a standard LLM pass — a phenomenon known as test-time scaling, or "long thinking."
How Do Tokens Drive AI Economics?
Throughout pretraining and post-training, tokens represent an investment in intelligence. During inference, they translate into costs and revenue. As AI apps become widespread, new economic principles are taking shape.
AI factories are designed to support high-volume inference, manufacturing intelligence for users by converting tokens into monetizable insights. Consequently, more AI services are valuing their products based on token consumption, offering pricing tied to input and output rates.
Some pricing plans give users a pool of tokens shared between input and output. For example, a user might spend a few tokens on a short text prompt to generate a long, detailed response. Alternatively, they might use most of their allowance to input a large document for summarization into a few points.
To manage traffic, some AI services also implement token limits, capping the maximum number of tokens a single user can generate per minute.
Tokens also define the user experience. Time to first token (the delay between a prompt and the start of the response) and inter-token latency (the speed of generating subsequent tokens) determine how users perceive the AI's performance.
There are trade-offs between these metrics, and the right balance depends on the use case.
For LLM chatbots, reducing time to first token helps maintain a natural conversational flow. Optimizing inter-token latency allows text models to write at a human reading speed or video models to hit specific frame rates. For models engaged in deep reasoning, the priority shifts to generating high-quality tokens, even if it takes longer.
Developers must balance these metrics to ensure a high-quality user experience while maximizing throughput — the total volume of tokens an AI factory can produce.
How to Achieve the Lowest Cost per Token
To overcome these challenges, NVIDIA’s full-stack AI platform provides a comprehensive suite of software, microservices, and blueprints, supported by robust accelerated computing infrastructure. This flexible, full-stack solution empowers enterprises to evolve, optimize, and scale AI factories for efficient token processing.
Grasping how to optimize token usage across different tasks allows developers, enterprises, and end users to extract maximum value from their AI applications.
Discover more about calculating the lowest cost per token and download the NVIDIA guide on Cost-Latency-Performance Optimization for AI Factories. Start building your AI factories on NVIDIA’s full-stack platform at build.nvidia.com.
