Eight experts. Who checks their work?

I keep giving increasingly elaborate computers the same very small job: explain what I just did before I forget why I did it.

In 2020, that meant fine-tuning GPT-2 on my commit messages. It learned my vocabulary, my punctuation, and my willingness to describe a week’s work as fix stuff. It did not learn to read a diff. A useful distinction, discovered after the training run.

Then GPT-3 won the rematch. This spring, LoRA made a personal set of habits small enough to email. The commit message remained one line long. Everything required to produce it kept getting more interesting.

Now Mistral has brought eight experts.

For a job I sometimes do with the word oops.

What the eight experts actually do

Mixtral 8×7B arrived this December with open weights under Apache 2.0, a 32K context window, and a sparse mixture-of-experts architecture. The name invites a lovely wrong picture: eight little engineers looking at my patch, arguing about the title, eventually asking the senior engineer to settle it.

There is no meeting.

At each layer, for each token, a router selects two of eight feedforward experts and combines their contributions. These are learned parts of one model, not eight chatbots writing separate answers. No expert gets assigned “security review.” None has a clipboard.

Mistral reports 46.7 billion parameters in total, with about 12.9 billion active per token. That is the attraction: more capacity without doing all of its computation for every token. It is not a built-in second opinion. The release description explains the routing, not an editorial process.

For the commit-message job, the relevant candidate is Mixtral Instruct. The base model and the instruction-following version are different choices; asking for a terse answer is a reasonable reason to choose the latter.

So the old experiment has a new contender. But I want to change how I judge it. “Which message would I keep?” sounded like a decent test when the bad answers were obviously ridiculous. It gets less useful when the wrong answer is the one I like reading.

A seven word promotion

Take this deliberately small patch:

 def fetch_manifest(url):
-    return requests.get(url, timeout=30).json()
+    return requests.get(url, timeout=5).json()

Here are three candidate messages, written to make the distinction obvious:

lower manifest request timeout to 5 seconds
fix slow manifest downloads
add retry handling for manifest timeouts

The first describes the edit. The second gives it a promotion. The third describes work nobody did.

Reducing a timeout does not make the remote server respond faster. It changes how long the client is prepared to wait under the timeout rules. There is also no retry in that patch, however much the word timeout might make one feel appropriate.

The annoying part is that the middle message sounds like a better commit message. It gives a reason. It has momentum. Somebody saw a problem and fixed it. Put it in a release note and a product manager might even nod.

The accurate version has merely changed a number.

That is the job, though. Sometimes we have merely changed a number.

Perhaps the real reason was a worker queue getting stuck. Perhaps it was a budget decision. Perhaps five seconds is too short and next week’s commit will put it back. None of those stories lives in these two lines. If I want the reason included, I need to supply it separately—and check that the patch actually supports the claim.

This is where I stop finding “sounds like me” flattering. I can already produce a confident explanation with insufficient evidence. I was hoping to outsource a different part of the process.

Testing the wrong answer I would accept

The rematch I want now has two scores, not one.

One is usefulness: would this save me any work? The other is unsupported claims: did the message say something the patch and supplied context don’t establish?

A message can score well on the first and badly on the second. That combination deserves its own red circle.

I’d keep the inputs identical across candidates: the same patch, the same short project context, the same examples. No feeding one model an entire repository and congratulating it for knowing more than the model given a single hunk. Where the input has to be shortened, record what was removed. Otherwise I’m comparing my preprocessing scripts and giving a neural network the trophy.

The cases shouldn’t all be dramatic fixes, either. Include a renamed variable. A deleted comment. A timeout reduced and then increased again. A patch containing two unrelated changes that should probably have been two commits. Ask for an honest description of boring work and see how much plot gets added.

The original human message isn’t sacred ground truth. It might be finally.

An ambitious benchmark: improve on a sigh.

Blind scoring still helps, but I would read the patch before revealing the candidate messages. Once a plausible explanation is in my head, it is surprisingly easy to look at the code and find the story I was just told.

A lint check cannot read my mind

Some requirements are easy to enforce. One line. No Markdown wrapper. No paragraph about how pleased the model is to assist with version control.

Those checks belong in code:

def subject_errors(text):
    subject = text.strip()
    errors = []
    if not subject:
        errors.append("empty subject")
    if len(subject.splitlines()) > 1:
        errors.append("more than one line")
    if len(subject) > 72:
        errors.append("too long")
    if "```" in subject:
        errors.append("markdown wrapper")
    return errors

Every candidate above passes this check. Including the imaginary retry implementation. The function is doing its job perfectly; its job just isn’t truth.

For review, I’d have the generator return an explanation beside the proposed subject: which changed lines support it, and which part needs context from the author. That explanation doesn’t go into Git automatically. It appears next to the patch, where I can disagree with it.

Checking that a quoted line really exists catches invented evidence. It doesn’t establish that the conclusion follows. The timeout line is real; “fix slow downloads” is still an interpretation it cannot support on its own.

There is an obvious temptation here: ask another model to review the first one.

I don’t object to that as an extra pair of eyes. It may catch something useful. But if both models see timeout and reach for the familiar story about intermittent failures, I now have two endorsements of the same guess. A second completion is not a test run, an incident ticket, or the person who made the change.

The useful review question is smaller than “is this correct?” It is: which claim in this sentence would you need more evidence to accept?

That question is useful when the reviewer is me, too.

The experts still need somewhere to live

There is a second appealing misunderstanding in the announcement. About 12.9 billion active parameters sounds like the storage problem has shrunk to match.

The other weights have not gone home. A later token or layer may need a different pair. Offloading can put weights in another kind of memory, but it cannot make them unnecessary.

Using the published, rounded parameter count, the raw arithmetic is:

46.7 billion × 2 bytes       ≈ 93.4 GB at 16 bits
46.7 billion × half a byte   ≈ 23.35 GB at 4 bits

Those are decimal gigabytes of idealized weight storage, not measured VRAM requirements. Quantization bookkeeping, tensors kept at higher precision, the attention cache, and runtime buffers still need space. “23.35 is less than 24” is not a deployment plan.

For this particular workload, I care less about a glamorous tokens-per-second number than about the pause between typing git commit and getting my editor back. The output is short. Reading the patch, loading the model if it isn’t warm, and moving data around may matter much more than generating the final few words.

And local doesn’t mean free. It means the API invoice is no longer where all the costs appear. There is hardware, memory reserved for a model that spends most of its day waiting for me, electricity, and whatever evening I donate when the runtime stops cooperating.

Keeping private code on hardware I control can be worth that. So can being able to keep a particular model version. Buying a graphics card to avoid typing a sentence is an unusual savings plan.

The cost I’d actually compare is cost per accepted, accurate message, including the time spent checking it. A cheap suggestion that needs detective work can be an expensive way to avoid typing seven words.

The next reader might be a model

Here is why this tiny task bothers me more than it used to.

A commit message doesn’t disappear after I click accept. It becomes part of the repository’s explanation of itself. Someone finds it six months later while investigating why the timeout is five seconds. They read fix slow manifest downloads and reasonably assume there was a slow-download problem, and that this change fixed it.

That someone might be another language model. Give it Git history as context and the earlier guess comes back wearing a date and a commit hash. It looks like a record now.

I have already built an on-call assistant that reads deployment history. So this isn’t a particularly long chain to imagine. One tool invents a motive to make a commit message nicer. Another tool retrieves the message to explain an incident. Somewhere in between, nobody remembers which bit was the invention.

The boring subject is a small kindness to that future reader. It tells them what changed and leaves the missing reason visibly missing.

Mixtral makes the generation side more interesting. Open weights make ownership more interesting. Neither changes who has to stand behind the sentence I put in the log.

Eight experts can help me write it.

I’m still the one pressing commit.