# Trying to compress text to spend fewer tokens

I tried six ways to make a big file cost fewer tokens when handing it to an LLM. I measured all of them and none worked. Here's what came out of each one.

Aug 8, 2026 · https://e1i0.com/en/intentando-comprimir-texto-para-gastar-menos-tokens.html

---

I wanted to bring down what it costs to hand big files to a model. It sounded easy: text repeats a lot, so it should be compressible.

I spent time trying ideas and measuring them. There were six, and none worked.

## First, what a token is

The model doesn't read letters or words.

It reads **tokens**, which are chunks of text it keeps in a list of 200 thousand. That list is public and you can go through it in [the tiktoken repository](https://github.com/openai/tiktoken).

And you get charged by the token.

Take this sentence:

`la esperanza es lo ultimo que se pierde`

That's 8 tokens. One per word.

So the question was: how do I make that cost less than 8?

## The six that didn't work

- **Sending it as binary.**

    The word `hola` is 1 token. In binary, `01101000 01101111 01101100 01100001`, it's 15.

    On a whole file, 7,721 tokens turned into 36,582. Five times worse.

    The model charges by the token, not by the byte, and a string of ones and zeros doesn't look like anything it has stored.

- **Hex and base64.**

    Same thing. `hola` in hex is `686f6c61`, which is 5 tokens.

    And in base64 it's `aG9sYQ==`, which is 7.

- **A dictionary of words.**

    The idea was to swap `la` for a short code like `§0`.

    But `la` already costs 1 token and `§0` costs 2, so the whole text came out 62% worse.

- **Gluing words together.**

    `la pelota` is 2 tokens. `la-pelota` is 4.

    I tried the 109 characters you can use to join them and none beats leaving the plain old space.

- **Looking for a token that carries two words.**

    I went through the model's entire list, the [200,019 tokens](https://github.com/openai/tiktoken/blob/main/tiktoken_ext/openai_public.py) it has stored. Not one has a space in the middle.

    The long tokens that do exist are things like 112 dashes in a row, worth 1 token but saying nothing.

- **Translating to another language.**

    `El cliente aceptó el producto` is 6 tokens. In Chinese, `客户接受了产品`, it's 4.

    It does save, but to translate you need a model, so you pay up front what you were going to save later.

There was one that did bring the tokens down: compressing the file with **gzip**, the same thing `.zip` files have always used. It cut them by 3 times.

But the model can't open a zip.

The only thing it knows how to do with what you send it is read it.

And that's the underlying problem: however cheap it comes out, if the model doesn't understand it, it's no use.

## Why language repeats so much

Read this:

> `El g_to se sub__ al te__do y no se qu_so ba__r.`

You understood it anyway. Letters are missing and it didn't matter, because your head fills them in on its own.

That's the redundancy of language, and it's the reason we understand each other.

It's what lets you read through typos, follow a thick accent, or hear someone in a bar with music playing.

Claude Shannon, the mathematician who laid the groundwork for how information travels, measured it in 1951 and put a number on it: [around 75%](https://www.princeton.edu/~wbialek/rome/refs/shannon_51.pdf).

And there's the catch, because that 75% is exactly what I wanted to remove:

- **If you take it out,** the file drops to a quarter of its size. But nobody can read it anymore, because reading is precisely filling in those gaps.
- **If you leave it,** it reads perfectly. But you pay for all of it.

That filler I saw as waste turns out to be what makes understanding possible.

My six ideas were all trying to remove that very thing, each in its own way, which is why they all hit the same wall.

## What does work is deleting

If you can't write it better, all that's left is saying less.

I tried [LLMLingua-2](https://github.com/microsoft/LLMLingua), from Microsoft, which runs on your laptop and is free.

What it does is go word by word deciding whether to keep it or drop it, and it hands back the ones that survived exactly as they were, without summarizing or rewriting anything.

Here's how it looks. This goes in:

> Of course, by now you and I know that the customer who has accepted and used the BUG products has been trapped into facing the influence of the reciprocity rule.

And this comes out:

> customer accepted BUG products reciprocity rule

From 33 tokens to 7.

Gone are the `of course`, the `you and I know`, the `who has`, the `the`s. What's left are the words that say what the sentence is about, and with those the model puts the rest together.

To find out whether it actually worked, I took two pages of a book and wrote down every concrete fact they carried. There were 12, things like:

- The 18% response rate a mail donation campaign got, and the 35% the same campaign got when it included a small gift.
- That the gift was address labels with the name printed on them.
- That the Jonestown massacre was in November 1978.
- The names: Diane Louie, Jim Jones, the anthropologist Marcel Mauss.

Then I compressed the text at different levels and counted how many of those 12 facts still showed up:

- **Compressed 2 times:** 11 of the 12 facts still show up.
- **Compressed 5 times:** still 11 facts.
- **Compressed 9 times:** 10 facts.
- **Compressed 17 times:** 8 facts.

The 5-times one is what surprised me, and there are two different things being measured there.

Compressing 5 times means **80% of the words** are gone. Out of every five words in the original, one is left.

But out of the **12 facts** only one disappeared.

So nearly all the text goes and almost none of the information.

And what it deletes is roughly that filler Shannon measured. It's removing what's predictable and betting the model will put it back, the same way you did with the cat sentence.

## How I tested it

In case you want to reproduce it, it's very little code.

```bash
pip install tiktoken llmlingua
```

To count tokens I use `tiktoken`, which is the tokenizer behind OpenAI's models. You give it text and it gives you back the list of tokens:

```python
import tiktoken

enc = tiktoken.get_encoding("o200k_base")

enc.encode("la esperanza es lo ultimo que se pierde")
# 8 tokens

[enc.decode([t]) for t in enc.encode("la esperanza es lo ultimo que se pierde")]
# ['la', ' esperanza', ' es', ' lo', ' ultimo', ' que', ' se', ' pierde']
```

The second one shows where each word gets split. That's where you can see the space is attached to the token.

And to compress, LLMLingua-2. The `rate` is how much you want to keep, so `0.25` means compressing 4 times:

```python
from llmlingua import PromptCompressor

compresor = PromptCompressor(
    model_name="microsoft/llmlingua-2-bert-base-multilingual-cased-meetingbank",
    use_llmlingua2=True,
    device_map="cpu",
)

resultado = compresor.compress_prompt(texto, rate=0.25)
print(resultado["compressed_prompt"])
```

The first time it downloads the model, a few hundred megabytes. After that, the two pages I used as an example went through in under a second.

## Two things I did find

- **Accents cost double.** `ultimo` is 1 token, `último` is 2. The tokenizer saw a lot of Spanish written without accents.

- **Notations really do compress.** "two hydrogen atoms and one oxygen atom" is `H2O`, four times less. But they only help where they apply. In a book there's barely anything to write in notation, in a financial report there's plenty.

## Where there is room left

Everything above I tested on written text, meaning sentences in a row: a book, an email, a contract.

With a spreadsheet, a CSV or a database export it's a different story.

In a 3,000-row table the same column carries the same value over and over, exactly the same. That one you can write once and then reference, without losing anything.

In written text that barely ever happens. Sentences resemble each other, but it's rare for two to be identical.

So the dictionary idea, the one that failed me with `la` and `que`, does work. Just in a spreadsheet, not in a book.

## What to use today

I didn't build anything new, but these three already exist and do the job:

- **[LLMLingua-2](https://github.com/microsoft/LLMLingua)** deletes words. You tell it how much to compress and it hands back the trimmed text.
- **[Headroom](https://pypi.org/project/headroom-ai/)** is the full version. It detects what kind of content it is, separates what has to be kept from what can be dropped, and saves the original in case you need it.
- **[graphify](https://github.com/3l1070r/graphify)** goes a different route. It turns a folder into a graph you can navigate, so instead of compressing the text you change what you send.

## What I take from it

The best part was measuring before programming. Each idea fell apart right away.

I didn't build what I wanted, but I came out understanding the problem a little better.

I didn't find a valid way to do it, but if something occurs to you, the idea is welcome.