I keep hearing people talk about tool calling, but I almost never hear people talk about Llogits.
And honestly, that’s a problem, because I don’t get them yet—at least not in the way I need to get them to build reliable systems.
What I do get is the goal: to imply determinism.
It can’t be forced because “forcing” suggests certainty, and certainty is not a thing we actually get with language models. We can shape probability and narrow the space of valid outputs, and we can build guardrails, but we’re still working with a model that is fundamentally generating the next token based on a distribution.
And in practice, what I care about isn’t deterministic content, but deterministic format.
Why I care about format more than I care about correctness
Lately I’ve been messing around with a side project* that does a lot with tool calling.
I’m building a self-hosted “background coding agent” that runs jobs, pushes branches, and opens PRs.
When your end goal is “make a PR,” you can’t treat the model like a chatbot. You need outputs that are actionable and parseable, and ideally outputs that fit a schema that your software can trust.
Tool calls are the most practical, mainstream way to do that today.
Tool calls are a way of telling the model to give you a certain kind of format, and that format is “perform this action.” I define what the action is, but it returns a format that calls that action, and that format should be as deterministic as possible.
And also: the dumber your model is, the less likely it is to be exact.
Tool calling, but in software engineer terms
If you haven’t built with tool calling yet, here’s the mental model:
Your app exposes tools (functions) with names and argument schemas
The model responds with a structured “call” instead of freeform text
Your code executes the tool, then feeds results back to the model
The model continues until it’s done
This is why I’m so into MCP.
MCP is basically a standard way to expose tools so you’re not rewriting the same tool interface for every model provider’s format. I don’t know what model I’m going to end up using long-term, so MCP lets me bet on a stable tool boundary while models churn underneath me.
Tool calling is basically a contract. It’s you saying: “If you want to do real work in my system, you have to talk to me in a shape my software can execute.”
So what are logits, and why am I even looking at them?
A logit is the model’s raw “score” for each possible next token before it turns those scores into probabilities and samples one.
So if tool calling is “tell the model what you want,” then logit manipulation is “change the odds right where the decision is made.” Grammars are the part that made this click for me.
Grammars are logit manipulation wearing a suit
When people say “logits” in the structured-output world, they often mean constrained decoding.At each generation step, the model produces scores for the entire vocabulary. Then the decoding logic masks out tokens that would violate the structure you want. The model can’t drift into invalid formatting, because the sampler won’t allow it.
Tool calling asks politely, grammars lock the doors.
It’s a distinction matters when you’re building automation.
Context-free grammars aren’t new (and that’s the point)
A context-free grammar (CFG) is a formal way to define what strings are valid in a language using production rules. This is the same family of ideas behind how we describe programming language syntax.
And yes, grammars have a history in voice systems too. The W3C Speech Recognition Grammar Specification (SRGS) exists specifically to define what patterns a recognizer should expect, using ABNF/XML grammar forms, so we’re not inventing a new concept.
We’re bringing formal language constraints back into the loop—except now the “thing generating strings” is a neural model.
So now that you have all this information, here is a summary from the guy that introduced me to logits, Chris Brousseau:
Logits and their manipulation are the only reliable way to offer control to LLMs. The difficulty they present is a shift in paradigm to figuring out what people will ask before they do, but that was always the difficult part of building anyway, and mastering it gives a significant advantage. Context-free Grammars (CFGs) provide an easy-to-understand framework for beginning this prediction methodology with very quick ROI for 2 reasons:
Structured outputs allow for easily implemented and enforced logit manipulation
It focuses the problem on what the LLM is good at: generating text, as opposed to what it’s bad at: verifying the validity or soundness of the text generated.
The failure mode that finally made this real for me
Here’s the story that turned “logits are interesting” into “oh, this is an engineering problem.”
I was experimenting with using logits/grammars instead of tool calling for a basic filesystem step: list a directory.
The tool was something like list_directory.
And the model kept calling list_directories.
One letter. One extra “s”.
The frustrating part is: the grammar was working. The model was returning the right shape. It was producing something that looked like a valid tool call, with the right fields, in the right places. It wasn’t rambling or mixing JSON with prose or ignoring the structure.
It was just… wrong by a character.
And this is where the “implied determinism” thing stops being philosophical and starts being operational because in my agent, I’ve given the model the ability to iterate.
That’s how most agents work, right?
You don’t call an agent and expect perfection in one pass. You call it and say:
complete this step
verify it’s complete
repeat until it’s actually done
then move on
Or you give it a workflow: steps one through five, repeat each step until it meets the definition of “done.”
That loop is the whole power of agents. And the whole danger.
When my model was looking for the incorrect directory, it started cycling. It would list. It would not see what it expected. It would list again. Then it would “correct” itself back into the same wrong tool name.
It wasn’t failing loudly or crashing, it was just doing what agents do: trying again.
And the result was basically an indefinite conditional that didn’t provide any value.
That was the moment I realized: reliability issues with structured output don’t just cause “bad answers.” They create bad control flow. They create loops.
Why this happens more with smaller models (and why logits still matter)
It’s important to note I was doing this on a 7B model. This is the part I think a lot of software engineers don’t internalize until they’ve built something: smaller models don’t do as well with tool calls.
Even when you hand them the schema or spell it out.
They’re more likely to get “close,” and “close” is a useless category when your software expects exact.
When I switched to the correct tool calling format for that model—meaning I matched the exact tool call shape it was trained to produce—I got better accuracy. But the model size was still a constraint.
That’s why logits are interesting here: logits / grammars can be a power tool for smaller models. Not because they magically make a 7B model smart, but because they tighten the space of outputs so the model doesn’t have to “remember” the format as well.
Tool calls are a contract, and grammars are enforcement.
And enforcement matters more when you’re running cheap models, local models, quantized models, and anything that isn’t a frontier model with a ton of instruction-following polish.
Now, I’m not claiming grammars solve everything (my failure case is literally proof they don’t). You can still be off by a letter if what you’re constraining is the shape and you haven’t constrained the vocabulary of tool names tightly enough.
But here’s the key lesson:
Tool calls can fail because the model drifts out of format.
Grammars can fail because the model stays in format but picks the wrong symbol inside that format.
That’s a different category of failure, and it’s one you can design around.
The difference that matters if you’re building agents
If you’re using models for chat, a weird output is annoying.
If you’re using models for automation, a weird output is a bug.
And bugs in automation don’t always look like exceptions. Sometimes they look like an agent confidently burning CPU in a loop.
So here’s my updated framing:
Tool calling is a really good default. It’s supported everywhere, it’s ergonomic, and it’s the easiest way to make models do actions.
Logits/grammars are what you reach for when the contract isn’t holding.
Especially when you’re on smaller models, or you’re trying to get deterministic format guarantees without paying for bigger inference.
And if you want a one-line “when should I care” rule:
If your agent ever gets stuck retrying the same step because it’s one character off from valid output, you don’t need a better prompt, you need enforcement.
When I wouldn’t reach for logits
This is the part I wish more people said out loud, because it makes the rest of the conversation more honest.
Grammars make format reliable. They do not make the model “correct.”
Constraints can degrade output quality if you make them too tight.
You still need validation, bounded retries, and sane stop conditions, because agents will happily loop forever if you let them.
So no, logits are not “the answer.”
They’re a reliability tool.
Sometimes the right move is still “use tool calling, use a larger model, and stop being clever.” But if you’re intentionally living in small-model land (local, cheap, self-hosted, quantized), logits start looking less like linguistics and more like engineering.
Where I think this goes
Most software engineers are not hanging out with linguists. We’re not thinking about formal grammars in day-to-day work. We’re thinking about interfaces, schemas, retries, and reliability.
Logits sound like math. Grammars sound like compilers. Most people tune out.
But the reason I’m forcing myself to learn this is simple:
When you move from “LLMs as chat” to “LLMs as software,” you eventually end up caring about the exact same things you’ve always cared about in production systems:
contracts
validation
bounded retries
failure modes that don’t silently waste time
Tool calling gets you surprisingly far.
Logit manipulation and grammar constraints are what you pick up when you’re tired of “surprisingly far” and you need the last mile of reliability.
And if you’re building agents that touch real systems (files, repos, CI, tickets, PRs) then that last mile is quite literally the whole job.
That’s why I keep saying “imply determinism.”Because I’m not trying to make a model correct, .
I’m trying to make it behave predictably enough that I can ship the software around it.
Stay Connected
Want to stay updated on what I’m working on? Here’s where you can find me: