💡 Giving AI smaller questions

A project produces a lot of information that should not become permanent documentation. A passing suggestion. A build that is still running. A decision nobody has actually accepted yet.

It also produces things worth keeping: why a tool behaves the way it does, which approach won, a preference that would save another round of explaining.

I keep reusable notes in a hub repository that coordinates work across my projects. Deciding what goes in there is a small judgment every time. Does this observation deserve a record, and does the evidence support what the record would say?

That is one of the places where I have started using Jev, TypeSafe’s model for focused judgments. The other experiments in this post are project names, finding shell commands and choosing blog stories.

All of them are early, and they all split the work the same way. Code gathers the evidence. Jev answers one bounded question. The caller decides what happens next. Generating names or writing prose is a job for a larger model.

flowchart TD
  A[Tools gather evidence] --> B[Jev answers a focused question]
  B --> C[Code and reviewer inspect the answer]
  C --> D[Take the next step]
  C --> E[Review or gather more evidence]
  E --> A

01🎯Give the model something it can answer

TypeSafe describes Jev as a “System One” model: it makes focused judgments on the context you give it and returns typed answers and probabilities rather than explanations or prose.

“Typed” means the answer has a defined shape. If I supply four categories, the result is one of those four. Code can use that directly without fishing a decision out of a paragraph.

The supplied context is called state. In my integrations, Jev cannot follow a repository link or fetch anything it is missing. The calling tool has to bring the evidence with it.

I use three question types. Choice picks from known options. Score rates something on explicitly ordered levels. Noul, despite the odd name, does something simple: it gives a probability for a yes/no question.

The distinctions matter. “Which candidate is best?” and “Is this candidate good enough?” are different questions. So are “How serious is this?” and “How likely is this to be serious?” A yes probability of 0.5 means uncertainty, not medium severity. The Noul documentation explains how to read it.

Independent questions can share the same state in one batch. They cannot see each other’s answers; my code combines them afterwards.

A tidy answer shape does not make the facts I supplied true. That part stays with me.

02🗒️Is this note worth keeping?

My knowledge-capture tool is a Rust command-line program. It takes a candidate note, its supporting evidence and the existing records, then asks Jev three questions: what kind of record is this, is it worth keeping, and does the evidence support it?

The categories are fact, decision, preference and skip. Code combines the three answers into capture, review or skip.

Even “capture” writes nothing. The command returns a recommendation. The coding assistant then re-reads the actual evidence and, where it matters, checks whether I accepted the decision before a record exists. The public capture workflow describes that handoff.

The integration is live, but I have not tuned the thresholds yet. In the first sanitized examples even the good candidates came back as “review”. That says the settings are cautious, not how accurate the model is.

I like that boundary. The model can flag a note worth looking at. Checking the evidence stays a separate step.

03🏷️Picking a project name

Naming a project splits the same way.

My naming workflow uses Codex to generate candidates. A Rust tool checks the relevant registries and APIs. Jev then rates naming fit, whether the supplied evidence points at a confusing product in the same category, and which candidate to recommend. “None” is always one of the options.

Jev does not generate the names here. It also cannot turn a failed registry request into an availability result. Unknown stays unknown.

In a live comparison of three candidates it preferred Taplume. Asking about absolute fit made that a lot less flattering:

I kept that result as it is. Winning a comparison is not the same as clearing the bar. A shortlist can have a first place and still not contain a name I should use.

Later web searches turned up historical listings for a similarly named Android app in third-party catalogs. Those were leads, not verified first-party evidence of a conflict. No name was accepted.

The tool gives me a shortlist. It does not give me legal clearance.

04🐚Finding a shell command

Quirl is my shell project. A shell interprets what you type into a terminal, including the flags that only become memorable after you have looked them up for the third time.

For a command-discovery experiment, I gave Jev descriptions of 62 known commands. It could choose one, or return NONE, CLARIFY or COMPOSE when nothing fit, when it needed more information, or when the request called for several commands.

It only selected options. It did not assemble command lines or run anything, and it is not wired into the shell.

One case asked for recursive directory sizes. The expected answer and Jev disagreed. Open the catalog evidence before deciding which one was wrong:

The catalog did contain options that supported directory totals. The expected answer was wrong, so I corrected it and kept the original next to the correction. The disagreement had found a bug in the test.

The benchmark report keeps the one unresolved case visible next to the validated results:

That is why I keep the evidence next to the recommendation. “The model disagreed” is the start of an investigation, and the command descriptions are what let me finish it.

05✍️Choosing a blog story

The draft behind this article went through one more experiment.

A new storytelling workflow gathers repository evidence and checks the existing posts and drafts. It hands Jev summarized story angles and coverage, then asks for a recommendation and a readiness rating per angle.

The first recommendation was useful. The benchmark disagreement made it into this article. It just needed a wider frame, because before a reader can care about Jev choosing a shell command, I have to explain what Jev is doing in my projects at all.

06🔍Keep the question small

The use I keep coming back to is a small decision between gathering information and acting on it. Is this note worth keeping? Does this name fit? Which command deserves a closer look? Which piece of work could become a story?

What these have in common is that I can put the evidence next to the answer and look at the disagreement. Where to set the cutoff for trusting an answer still needs testing. A confidence value says how concentrated the model’s answer is. It cannot tell me the question was a good one.

For the next integration I would start the same way: one recurring judgment, the context it needs, and a way to leave it unresolved. A name can stay unchosen. A note can wait for review. And a perfectly reasonable story recommendation can turn into a different article.