I trained a tiny LLM to choose bigger LLMs

It reads a prompt, predicts which bigger model can handle it, and gets out of the way.

I built Succinct to answer the question in How small a model can I ship?: take a real workload, compare a model ladder, and choose the smallest model that clears the quality bar. It is a useful decision, but a fairly static one. You test, choose a model, and put it into production.

Production is rarely one fixed kind of request. One call might extract a field, another needs strict JSON, another uses tools, and another needs more reasoning. What if I wanted to choose between OpenAI models for each request as it arrived?

That became the next piece in the Succinct family. I trained a tiny model to read the incoming prompt and predict whether Luna, Terra, or Sol is the cheapest configuration likely to pass.

A model router, basically.

I had never trained a language model from scratch before. This was mostly a learning project, and I wanted to understand the whole path: generating data, training, testing, exporting, and preparing it to run locally on my Mac.

The obvious label is kind of wrong

My first thought was to ask a big model to generate prompts and label them as easy, medium, or hard.

Extract the invoice number from this email.
Difficulty: easy
Use small model.

The problem is that “easy” is not a real label.

Easy for who? A 3B model? A 7B model? A model with tools? A model running with a strict JSON schema? The answer changes all the time.

A big model can have an opinion about whether a task is easy. That does not mean the smaller model I am actually using can do it.

So I used a different approach. I generated the tasks first, then ran every task through the actual candidate ladder: GPT-5.6 Luna with reasoning disabled, Terra with low reasoning, and Sol with medium reasoning. The label became the cheapest complete configuration that really passed.

That feels much more honest.

Making fake data that is still useful

The data was synthetic. I built it from hidden task specifications instead of asking a model to make up a prompt and then make up an answer.

{
  "family": "extraction",
  "variables": {
    "text": "Hey, Maria Lopes here. The €39.99 order was wrong. Could you arrange a refund?"
  },
  "ground_truth": {
    "name": "Maria Lopes",
    "amount": 39.99,
    "action": "refund"
  }
}

Then I rendered it as a task for the candidate model:

Extract name, amount, and action as JSON.

Hey, Maria Lopes here. The €39.99 order was wrong. Could you arrange a refund?

Now I already know the correct answer. The model needs to find:

{
  "name": "Maria Lopes",
  "amount": 39.99,
  "action": "refund"
}

This meant I did not need another AI model to decide whether an answer looked good. For many tasks I could check the answer in code.

I generated twelve task families around extraction, classification, JSON output, maths, tool choice, policy lookup, retrieval, SQL, structured summaries, PII detection, multilingual prompts, and conflicting instructions. Each hidden specification produced a clean prompt and a messy version with small typos. The larger pretraining corpus also included terse, verbose, and adversarial variants. Some tasks contain distractors or quoted instructions that try to break the output format.

The clean prompt is rarely the real prompt.

Then came the expensive bit

For each generated task, I sent the prompt to every candidate model and graded each answer. If Luna passed, great. If Luna failed but Terra passed, Terra became the route. If only Sol passed, that was the label.

I ended up with 10,000 routing examples and 30,000 logical model evaluations. OpenRouter reported $30.63 in incremental cost for the completed run, with zero candidate API errors.

That was honestly better than I expected. It is still enough work that I would not repeat the experiment casually. For a first benchmark around a product task, it felt reasonable.

{
  "prompt": "Extract the refund reason and requested resolution...",
  "results": {
    "small":  { "passed": false },
    "medium": { "passed": true },
    "large":  { "passed": true }
  },
  "best_route": "medium"
}

This is the bit I find most interesting. The router is learning the behaviour of the models I actually want to use, under the exact settings I tested. Change the model, reasoning effort, schema, or decoding settings and the old labels are no longer trustworthy.

I trained two versions

First I trained a very small model with 919,430 parameters. It was there to prove that I understood the pipeline. Could I generate labels? Could the model learn them? Could I test it properly? Could I avoid building a huge mess before I knew whether the idea worked?

That tiny version memorised the 1,000-example training slice well enough to reach 98% exact pass-vector accuracy after 1,100 steps. Pipeline working.

Then I trained the real prototype. It is still tiny by modern standards: 13,769,862 parameters. This is not a model you would chat with. It reads a prompt and predicts three pass probabilities, one for each candidate configuration. The routing policy picks the cheapest one above a calibrated safety threshold.

Before supervised routing training, I trained one copy on 100,000 generated prompts so it could learn the basic shape of the text. I trained another copy from random initialisation.

The pretrained version did slightly better: 92.44% validation pass-vector accuracy versus 92.18% from scratch. No dramatic magic. Just better enough to keep.

Did it work?

On the frozen synthetic test set, the final router picked the exact empirical route 95.09% of the time. It made an unsafe downroute 0.55% of the time, and the selected model passed 99.45% of tasks. Across the full benchmark, the downstream calls cost 61.6% less than always using Sol.

I expected the small router to save money. The surprise was that it was also safer than the much bigger models I asked to make the same decision.

There is a humbling detail here. A TF-IDF logistic regression baseline reached 94.78% exact route accuracy. The transformer beat it on exactness, safety, pass rate, and savings, but only narrowly. For a narrow classification problem, the boring baseline deserves respect.

Against bigger routers

I also asked five much larger models to act as zero-shot routers. Each one saw the same 1,628 frozen prompts and chose between Luna, Terra, and Sol without seeing any training examples.

The router has 13.8M parameters. Its only job is to look at a task and decide whether Luna, Terra, or Sol is the cheapest model that can do it well enough.

It tied Claude Sonnet 5 for exact route accuracy. The route label is still a proxy. The thing I actually care about is whether the model it selected completed the task.

Sonnet picked the exact cheapest route 95.09% of the time too. Its selected model passed 95.64% of the time, compared with 99.45% for Succinct. Matching the ideal route label is clearly not the whole story.

The result I kept coming back to was Sol. It was the strongest model in the group when it did the work itself. When I asked it to choose which model should do the work, it selected a passing model only 69.10% of the time. Being good at a task does not automatically make a model good at deciding who else should do it.

The failure mode

An exact-route mistake can go in two directions. An overroute spends more than necessary. An unsafe downroute chooses a cheaper model that fails. The second one is the failure I care about.

Succinct's unsafe downroute rate was 0.55%. The nearest large-model judge was Claude Sonnet 5 at 4.36%.

The overhead

A router sits in front of every request. If it needs another large remote model, its latency and bill arrive before the selected model starts answering. Succinct made its decision locally in 26 milliseconds. The five remote judges took between 7.77 and 9.57 seconds.

Opus showed the other side of this. It was reasonably accurate as a router, but asking it on every request was so expensive that the full system cost more than sending every task straight to Sol. A smart routing decision is not useful if making it costs too much.

The practical result

The routing decision only matters if the whole request gets cheaper. Always sending the frozen test set to Sol cost $4.41. Succinct's selected downstream calls cost $1.69. That is a 61.6% saving, while the task pass rate moved by only about 0.06 percentage points.

How I measured it

The benchmark starts with hidden task specifications. I run the real candidate models, grade their answers mechanically whenever possible, and label each prompt with the cheapest configuration that passed. The zero-shot judges and Succinct then route the same frozen prompts.

The boundary

Those are the nice-looking results. There is also a big warning: the prompts are synthetic.

The tasks are messy on purpose, and the labels are empirical because I actually ran the candidate models and graded their outputs. That makes them more useful than one model guessing what another model might find hard. Synthetic prompts are still not real user traffic. Real users will be weirder than a benchmark I made for myself.

I would not put this in front of production traffic and say “done”. The next honest step is to collect a few hundred anonymised prompts from the actual product and see whether the synthetic benchmark predicts what happens in the real world. If it does not, that gap tells me what the synthetic data is missing.

Exporting it to MLX

The last goal was to package the router for my Mac. I trained with PyTorch on Linux, then exported the weights to a standalone MLX bundle.

The bundle needs more than the weights. It includes the tokenizer, model config, candidate labels, calibration threshold, and a small runtime that makes the same routing decision as the training code.

The float32 weights are about 55 MB. PyTorch and the MLX CPU runtime matched across the frozen parity examples, with a maximum pass-logit difference of 0.00000381. I still need to measure Metal latency on Apple Silicon, so that last performance number stays blank for now.

The model itself is now public on Hugging Face, along with the MLX runtime and calibration files.

What I learned

The hard part was making labels I trust.

A small classifier can learn this routing job pretty well. If the benchmark is fake in the wrong way, though, it will learn fake confidence too.

I do not think the tiny router is smarter than Sonnet or Opus. It has learned one narrow thing: the real pass and fail behaviour of this exact Luna, Terra, and Sol setup, using this prompt format and this quality bar. That focus is the advantage.

The better loop is:

1. Generate tasks from hidden specifications.
2. Run the real candidate models.
3. Check the answers mechanically whenever possible.
4. Label each task with the cheapest configuration that passed.
5. Keep testing against real work.

This project is still a prototype for one exact setup.

Still, it feels like a useful proof. For one narrow routing problem, a very small specialised model was faster, cheaper, and safer than asking a frontier model to make the decision every time. It has a real model, a public dataset, actual cost numbers, a reproducible training setup, and an MLX export. That is enough for me to keep going.