Understanding Tokens in AI Studio for HubSpot
What Are Tokens?
Tokens are essentially small chunks of text that AI models use when processing and generating language. You can think of them as pieces of words. When you input text into an AI model, the text is broken down into tokens before it’s processed.
Here are a few common rules of thumb (adapted from OpenAI’s documentation) to help you gauge token usage:
• 1 token ≈ 4 characters in English
• 1 token ≈ ¾ of a word
• 100 tokens ≈ 75 words
• 1-2 sentences ≈ 30 tokens
• 1 paragraph ≈ 100 tokens
• 1,500 words ≈ 2,048 tokens
Examples
• Wayne Gretzky’s quote “You miss 100% of the shots you don’t take” contains 11 tokens.
• The U.S. Declaration of Independence transcript has 1,695 tokens in English.
Language Variations
Tokenization varies by language. For example, the Spanish phrase “Cómo estás” contains 5 tokens for just 10 characters, resulting in a higher token-to-character ratio than typical English text.
Further Exploration
If you want to see exactly how tokens are counted:
• OpenAI Tokenizer Tool: Interactively calculate and visualize tokens.
• Tiktoken: A fast BPE tokenizer to programmatically tokenize text for OpenAI models.
How AI Studio for HubSpot Counts Tokens
Token Allocation in Your Plan
Each AI Studio plan has a base number of tokens included. For example, the Pro plan includes 1,000,000 tokens per month. This is your core token allowance before any multipliers from specific models come into play.
Model Multipliers
In AI Studio, different AI models have different cost structures—reflected as multipliers. This means:
• OpenAI GPT-4o mini: Multiplier 0.05
• Using this model gives you more total tokens.
• On the Pro plan (1,000,000 tokens base), a multiplier of 0.05 effectively lets you use 20,000,000 tokens of GPT-4o mini per month (because 1,000,000 ÷ 0.05 = 20,000,000).
• OpenAI o1: Multiplier 5
• This model is more expensive, so you’ll have fewer tokens to spend.
• On the Pro plan, you’ll be able to use 200,000 tokens of o1 per month (1,000,000 ÷ 5 = 200,000).
The higher the multiplier, the more expensive the model, resulting in fewer tokens for that specific model within your plan’s monthly allowance.
WebSearch, Company Insights (website parser)
Some workflow actions has a flat rate for one execution, for example Company Insights consumes 500 tokens for one execution.
Choosing the Right Model
Cost-Effective, High-Efficiency Models
For processing CRM data or handling large volumes of day-to-day tasks, we recommend these cost-effective models:
• OpenAI GPT-4o mini
• Google Gemini Flash 2.0
Both have a token multiplier of 0.05, making them an excellent choice for bulk operations due to their low cost in tokens.
Advanced, Premium Models
For more complex tasks such as in-depth content generation, creative writing, or high-level problem-solving, consider:
• OpenAI GPT-4o
• Google Gemini Pro
• Anthropic Claude (particularly good for content like email drafts, articles, etc.)
While these models are more powerful, they have higher multipliers and thus fewer tokens are available per month relative to your base plan allowance