# What if your tests need testing?

A technique from 1971 sat unused for fifty years because its bottleneck was human judgement at scale. That is exactly what changed.

Aug 16, 2026 · https://e1i0.com/en/tus-pruebas-necesitan-ser-probadas.html

---

It always goes the same way. You write tests, coverage goes up, the dashboard turns green, and at some point you wonder whether those tests are any good or they are just giving you a sense of safety.

I wanted to answer that for myself. Code gets written and reviewed much faster now with a model beside you, and the hard part became knowing whether what comes out of there is any good.

You have this function:

```python
def suma(a, b):
    return a + b
```

And this test:

```python
assert suma(2, 2) == 4
```

It passes. 100% coverage.

Now change the `+` to a `*`. `2 * 2 = 4`. **The test still passes.**

Your suite cannot tell addition from multiplication. Or from exponentiation, since `2² = 4`.

## Who watches the watchmen

A test claims something: *"if the code stops doing X, I fail"*. That claim almost never gets verified.

It is a smoke detector. You install it and it stays quiet. The silence can mean there is no fire or that the battery is dead, and from the outside they look the same. To find out you press the test button, which starts a fake fire on purpose.

## Mutation testing is that button

The idea was [proposed by Richard Lipton in 1971](https://en.wikipedia.org/wiki/Mutation_testing), while still a student, and DeMillo, Lipton and Sayward developed and published it in 1978. It is mechanical: the tool breaks one single thing in your code, runs your suite, and notes whether anybody noticed.

![The mutation loop: one thing gets broken, the suite runs, and the mutant ends up killed or alive](https://e1i0.com/assets/images/mutation-testing-ciclo-en.png)

Each change is a mutant. The function above generates six, one per operator that can replace the `+`.

![The six mutants of suma(a, b) against two test cases: with suma(2, 2) the product and the power survive, with suma(2, 3) all of them die](https://e1i0.com/assets/images/mutation-testing-suma-en.png)

## Why it sat unused for fifty years

Run the tool on a real file and it hands you six hundred survivors, each one with no explanation. That is where the heavy work starts, because they have to be read one by one and sorted into a missing test, dead code, or a change that does not alter behaviour and can be left alive. That is days of a person for each module, and I suspect that is a good part of why the technique stayed in academia for so long.

**And that is what changed.** I ran ~3,400 mutants across six modules of [Frauddi](https://frauddi.com) in an afternoon. I handed the results to a model and it grouped them for me: which ones pointed at the same thing, which I could ignore, what to look at first. The judgement was mine, but getting to what mattered went from days to minutes.

## What it found

Tests that did not check what I thought they checked. Three examples, all of them mine.

**The test that expects the default.** If a function ends up returning a default value and your test expects exactly that value, break whatever you like in there and the test passes all the same.

```python
def nivel(puntos):
    if puntos > 100:
        return "oro"
    if puntos > 50:
        return "plata"
    return "bronce"

assert nivel(10) == "bronce"
```

That test only walks the last line. Change the `>` to `>=` and it stays green.

**The limit nobody was checking.** I wrote 54 cases for a detector. My two cases for one signal sat so far from the limit that triggers it that neither one was checking it.

```python
def es_sospechoso(ratio):
    return ratio > 0.20

assert es_sospechoso(0.71) is True
assert es_sospechoso(0.02) is False
```

Move that `0.20` to `0.50` and both tests still pass. I could put it anywhere with all 54 cases green.

**The operation that made no difference.** A line adds two numbers, and in every test of mine one of them was zero.

```python
def posicion(inicio, n):
    return inicio + n

assert posicion(0, 3) == 3
```

With `inicio` at zero, swapping that `+` for `|` or `^` gives the same result. It is the `suma(2, 2)` from the top of this post, in production and written by me.

That is where the one rule I keep came from: **when a maths operation survives, check whether your test case makes it irrelevant.**

## The signal that helped me most

A function where **not a single mutant dies**: you can break the whole thing and nobody complains. It showed up three times and all three had something to clean up:

- A function nobody called any more.
- Another that the test ran without checking anything it returned.
- A third that always got the same value.

Those tests were written by a model and I signed off on them in review. They read fine and coverage went up, and neither of us noticed they were not checking anything.

## After the first pass

That was the first pass. Later I extended the exercise to the rest of the application, with [end-to-end tests](https://en.wikipedia.org/wiki/System_testing), [fuzzing](https://en.wikipedia.org/wiki/Fuzzing), [unit](https://en.wikipedia.org/wiki/Unit_testing) and [integration](https://en.wikipedia.org/wiki/Integration_testing) tests. Of all the techniques, mutation testing surfaced the most.

## When it is worth it

It is expensive in compute. Every mutant is a full run of the suite, and it takes thousands for a handful to show up. It does not go in CI, and I did not run it over all the code.

And it has an order. First you raise coverage, then you measure whether it helped. A line with no coverage has all its mutants alive by definition, so measuring it tells you nothing.

## What this says about LLMs

There is a lot of talk about models handing us new techniques. Something else happened here: **a fifty-year-old technique became practical without changing at all.**

It is worth looking back with that in mind. There are ideas that got shelved because they needed too many hours of somebody reading and sorting, and that is exactly the part where a model can help you a lot.

If you are curious, take one module, just one, from the code you care about most, and run mutation on it. What comes out will tell you more about your tests than any coverage percentage.