What AI Tokens Are and How AI Costs Are Calculated

Why does my bill keep going up?

I’m a software engineer and the founder of Workflow Autopilot. I’ve spent thousands in AI spend across the companies that I’ve worked with, which are small to medium-sized businesses.

What a token is

Token is essentially a way to break down English language into smaller bits. So for more complicated words, you’ll have many tokens to represent that word. And those tokens are fed into the AI.

One sentence, usually you can ballpark it around 25 or so tokens, versus a five page document being 2,500. So 2,500 tokens on average.

Input tokens

So what are input tokens and what are output tokens?

Input tokens are simply what gets fed into the AI. It’s the prompt. It’s what you type into the chat window. And it’s also the accumulation of the conversation that you’ve had up to that point. And any documents that you may have attached.

So examples of the input tokens are documents that you put in, like contracts you want summarized, information about a client or a potential prospect. Anything you put in the chat window for actually instructing the AI to do what you want, as well as the conversation that you’ve had up to that point, will all be going into the input tokens.

How AI cost is calculated

How AI cost is calculated is it’s calculated by a hybrid combination of input tokens and output tokens. So they’re not always weighted the same or cost the same.

So for instance, you might have a $1.20 per million input tokens, but $4.20 per million output tokens. I’d say in almost all cases, it’s true that the output tokens are going to be way more expensive than the input tokens.

Why output tokens cost more

Without getting too complicated, input tokens are able to be calculated all at the same time. You’re providing all of the tokens at one point and that’s the prompt that’s sent off to the AI.

Meanwhile the output tokens are generated one at a time. If you’ve ever used an agent you’ll see that it’s constructing the sentence as it gives you the output and that’s because it’s literally calculating each character, each token as it goes because it needs to rely on the previous tokens in order to generate the next token in the list.

So as far as processing power, it’s a lot more complex and takes a lot more power to actually generate the output than it does to read the input. And so that’s mainly the reason why the output tokens are more expensive than the input tokens.

What this means for your bill

Essentially input tokens are cheaper than output tokens. Since this is a cheaper token, it’s way better to front load all your tokens into the input.