The small model gets a smaller job

My plan for Phi-2 involved three boxes: network, timeout, and deployment.

There is a problem with those boxes that does not require a neural network to discover. A timeout is a symptom. A deployment is an event. A network is a large collection of things I blame when I don’t yet know what happened.

Naturally, I wanted a model to choose exactly one.

This is the part of building with AI that doesn’t make the demo: sometimes the machine isn’t failing to understand your categories. You haven’t finished inventing them.

A smaller candidate for the old job

After last month’s excursion into Mixtral, I wanted to go in the opposite direction. Fewer parameters. Less machinery. Preferably a task small enough that I could explain what success looked like without pointing at a leaderboard.

Microsoft’s Phi-2 release made that tempting: 2.7 billion parameters, trained with an emphasis on carefully selected web material and synthetic educational text. The interesting proposition wasn’t simply that a smaller model could imitate a larger one. It was that what you teach it deserves as much attention as how much room you give it to learn.

There is an important catch: Phi-2 is a base model, not an instruction-tuned assistant. It hasn’t had instruction fine-tuning or reinforcement learning from human feedback—the sort of post-training commonly used to make models behave like the conversational employees we now expect to hire by API call.

That doesn’t mean it cannot answer a question. It means “be helpful, but only in this very particular way” isn’t a free feature I should assume.

The old job was my on-call briefing assistant: gather incident context, propose a diagnosis, attach evidence, hand the whole thing to a human. Useful ambition. Also a surprisingly large job once you unpack the word diagnosis.

Before replacing the model, I wanted to see whether I could replace the job description.

Three labels and no agreement

Consider this little incident sketch:

08:41:58 INFO  worker starting after deployment
08:42:01 WARN  queue subscription closed; reconnect pending
08:42:03 ERROR export task deadline exceeded

Is that a deployment incident, a network incident, or a timeout?

The timeout is there in the text. The deployment is there too. A network problem might explain the closed subscription, but so might a configuration mistake, a server restart, or something outside this tiny excerpt. Choosing one box quietly requires deciding which question the boxes are answering.

Are we labeling the symptom? Guessing the cause? Choosing the person to wake up?

Those are different products. I had put them in one dropdown.

It gets worse if I turn these examples into a fine-tuning dataset. Today I label this timeout because that is what the alert says. Tomorrow I label the same shape deployment because I know what the postmortem found. The model sees inconsistent targets. I see a disappointing accuracy number and start shopping for more parameters.

A very expensive way to avoid writing a definition.

The immediate fix could be a better taxonomy: separate fields for observed symptom, suspected cause, and ownership. But I don’t actually need all three fields to make the first minute of an incident less unpleasant.

I need somewhere useful to look.

Let it point instead

So the smaller job is this: given a reported symptom and a short list of events, nominate one event worth inspecting first. Return its ID. If the supplied evidence doesn’t support a useful choice, return NONE.

No diagnosis. No severity score. No proposed shell command.

Here is a complete example prompt, with the answer deliberately left for the model:

Instruct: Pick the event a person should inspect first for
the reported symptom. Return one event ID, or NONE if the
events do not distinguish a useful starting point.

Symptom: Export jobs wait in the queue without starting.

E1 08:41 worker: heartbeat accepted
E2 08:42 queue: 18 jobs waiting
E3 08:42 worker: task subscription closed; reconnect pending
E4 08:42 api: export request accepted

Output:

For this example, I’d start at E3. E2 repeats the symptom; E3 suggests a concrete next question: why isn’t the worker subscribed? It doesn’t establish the root cause, and I wouldn’t want the interface to pretend it did.

The Instruct: ... Output: wrapper comes from Microsoft’s model card. The word Instruct in a prompt does not magically make the underlying model instruction-tuned. It is a suggested text format, not a change to its training.

The result should behave like a colleague putting a finger on the screen: “Have you looked at this?”

That is a much more modest product than a synthetic incident commander. It is also closer to what I want when I’m tired.

The list is part of the model

Phi-2’s 2,048-token context window means the input cannot be a bucket of logs tipped into a prompt. Instructions, examples, events, and the generated answer all have to share the space. An event ID is short, but it isn’t necessarily one token.

My older assistant already needed log compression. Here the compression makes a different promise: preserve useful places to investigate, not enough material to write a convincing explanation.

That changes some choices. Keep event IDs stable so a suggestion can link back to the original record. Collapse repeated messages while retaining their count and time range. Keep the original logs available. Remove credentials before anything reaches the model, even if the experiment runs locally.

Most importantly, don’t quietly chop off whichever end fails to fit. Reject an oversized input or choose a smaller, explicit window. A model asked to pick from four events cannot nominate the fifth event my preprocessing removed.

And please don’t select only ERROR lines. In the example above, the useful starting point is a warning. Logging levels describe what the author of that particular line thought was worth shouting about. They are not a universal ranking of investigative value.

By this point, “the model picked the wrong event” is already an incomplete bug report. Which events did I let it see?

A short answer can still waste your evening

The mechanical boundary can stay small:

def event_from_reply(reply, events):
    choice = reply.strip()
    if choice == "NONE":
        return None
    if choice not in events:
        raise ValueError("reply is not a listed event ID")
    return events[choice]

That rejects an invented ID or a paragraph where an ID should be. It does not tell me whether E3 was a good choice. Picking a real but irrelevant event is entirely legal Python.

The interface matters as much as this check. Highlight the suggestion, but leave the whole event list visible. Don’t hide the other lines behind “AI has analyzed this incident.” I know how quickly a convenient default can become the only thing anyone reads.

NONE needs care too. It is permission to decline, not proof that the model knows when it is wrong. A model that always chooses NONE has achieved perfect avoidance of embarrassing suggestions and provided no service whatsoever.

For evaluation, I’d start with three cheap opponents: chronological order, the first error, and a keyword search based on the alert. If the model can’t improve on those, it is a complicated bookmark.

The question isn’t whether its answer matches my favorite line. Several starting points may be reasonable. The useful outcome is whether a person reaches relevant evidence sooner, without being steered away from a better lead.

That requires keeping track of suggestions that help, suggestions that distract, and cases where the model declines. One accuracy percentage would hide too much of the story.

Annoy it before trusting it

The tests I like for this design are almost petty.

Duplicate the timeout message twenty times while leaving the subscription warning once. Does the pointer chase repetition? If I deduplicate first, do I preserve the fact that one event happened twenty times?

Shuffle the event order in the prompt without changing timestamps or IDs. If the suggestion follows whichever line is now last, I have learned something useful about the system—and it isn’t about the incident.

Add a frightening but unrelated error from another service. Then add an ordinary-looking event that actually changes the interpretation. The word FATAL should not win a staring contest against relevance automatically.

Finally, remove the useful clue. A test set that always contains a convenient answer teaches a thoroughly misleading lesson about incident response. Sometimes the logs are insufficient. Sometimes the important service didn’t log the thing. Sometimes the next step is to collect different evidence, not squeeze harder on the evidence already available.

These tests wouldn’t prove Phi-2 ready for production. They would tell me whether this narrow interface is worth pursuing. That is enough for an experiment; it doesn’t need a victory parade.

What got smaller

The appealing thing about a small model is easy to describe: less to load, less hardware pressure, a chance to keep a useful task close to the machine where the data already lives. Whether it is fast enough still depends on the actual hardware and runtime. “Small” is not a latency measurement.

The less obvious benefit is that it makes a vague assignment harder to hide.

A bigger model might write a beautiful paragraph connecting a deployment, a lost subscription, and a timeout. It might even be right. But the paragraph can let me postpone deciding what the tool is supposed to do for its user.

Designing for one event ID forces the decision. This tool is for choosing where to look next. It isn’t for announcing what happened. Once that is explicit, the input, the interface, and the evaluation start making sense together.

The three original boxes can go. I don’t need to teach a model the difference between a symptom, a suspected cause, and my organizational chart before it can point at an interesting warning.

I still need someone to investigate.

I’d just like them to start one useful line closer.