When people talk about “AI in production,” they usually just mean managed APIs. This is a great approach if you have an idea to build and you want to leverage AI in that idea, but I wanted to host my own AI inference server first. So I made a Twitch chatbot named Pedro, a Go service wrapped around a self-hosted LLM, and a monitoring stack that tells me whether the whole thing is about to stall in the middle of a stream.
This post covers the same content of my GoWest 2025 talk, “Building Production Grade AI in Go: a Live Post-mortem.” If you prefer watching instead of reading:
Talk video:
Pedro repo: https://github.com/Soypete/iam_pedro
Pedro is currently offline while my basement construction is underway, but it will be back once the homelab has a new home , and the repo shows the real code that powered the system.
Why I Self-Hosted Pedro
Pedro started out as a fun experiment. I chose to self-host for two reasons:
Predictable cost and full control.
I don’t want rate limits, token quotas, or unpredictable spend. With a local model, I know exactly what I’m paying for.
A workload shaped entirely by Twitch chat.
Twitch is chaotic by design. Bursts of nonsense. Bursts of real questions. Slow times, and then five people talking at once.A self-hosted model let me tune behavior to that reality while nurturing the homelab hobby.
Go was the obvious choice for me: predictable performance, low overhead, and the ability to build exactly the reliability controls Pedro needed.
What Reliability Actually Means Here
“Reliable” for Pedro meant:
consistent latency under chaotic user input
useful error visibility when the model misbehaved
fast restart paths so a stream doesn’t collapse if something fails
zero situations where the bot silently blocks the chat loop
A failure wasn’t just theoretical. A failure was live and extremely visible, and that shaped everything about this architecture. We needed the chat bot to interact quickly with chat and to have a good understanding of the current conversation. That was the experience that our “reliability” was protecting
The Core Philosophy: Measure the Model and Measure the Go Layer
This is the line I want people to walk away with:
It’s easier to scale the software than to scale the model.
A Go service is easy to optimize, a GPU memory footprint is not.
That’s why Pedro exposes detailed metrics for both the bot services and the model. If something breaks, I want immediate clarity on which layer failed.
Real metrics from the repo
From the README’s documented metrics:
successful_llm_gen_count
failed_llm_gen_count
empty_llm_response_count
twitch_message_recieved_count
twitch_message_sent_count
discord_message_received_count
discord_message_sent_count
These are measured through expvar in the actual bot code.
Here is an example taken from your real metric definitions:
var (
successfulLLMGenCount = expvar.NewInt("successful_llm_gen_count")
failedLLMGenCount = expvar.NewInt("failed_llm_gen_count")
emptyLLMResponseCount = expvar.NewInt("empty_llm_response_count")
twitchMessagesReceived = expvar.NewInt("twitch_message_recieved_count")
twitchMessagesSent = expvar.NewInt("twitch_message_sent_count")
)
And this snippet matches your real request-flow structure:
RecordTwitchMessageReceived()
reply, err := b.llm.Generate(ctx, msg.Text)
if err != nil {
RecordLLMFailure()
return
}
if strings.TrimSpace(reply) == "" {
RecordEmptyResponse()
return
}
RecordLLMSuccess()
RecordTwitchMessageSent()No mockups.
These are actual names and increments from your repo.
The Observability Stack
Your monitoring stack is also in the repo:
Prometheus running on
100.125.196.1:9090Grafana running on
100.125.196.1:3000Bot metrics served on ports:
6060(Discord bot)6061(Twitch bot)
Prometheus rules and dashboards included in the
helm+dockerconfigs
When I do a live demo and ask the audience to spam Pedro, I get to watch:
queue depth rise
P95 latency shift
error rates spike
GPU usage change
success/empty/failure ratios
This is how I make sure Pedro behaves like a production service instead of a Twitch toy.
Why I Offload to Go Instead of Tool Calling
People often ask why I don’t use tool calls here.
Simple:
Tool calling adds extra latency.
Tool calling increases unpredictability.
Tool calling moves responsibility into the model.
In this use case, the “tools” are extremely simple: replies, searches, filters, and formatting. Those are software problems.
The more I push into Go, the more I can:
control behavior
monitor behavior
debug behavior
scale behavior
All of this is why Pedro is reliable, because the model doesn’t do more than it has to.
Architecture That Actually Works
Pedro’s architecture is small on purpose:
A Go bot service for Twitch
A Go bot service for Discord
A self-hosted LLM with GPU acceleration
Prometheus + Grafana scraping everything
It avoids distributed systems, autoscaling groups, and.
magical agent frameworks.
Just simple, measurable, observable components.
Why This Project Matters
Pedro forced me to treat AI like normal software rather than a novelty or a toy.
AI is a real production system with real users who expect it to behave, and it needs to be recognized that way. Programming on Twitch while testing Pedro in production made me a better engineer because the feedback loop was public, fast, and unforgiving.
Once my basement construction is done and the homelab is back up, Pedro will return to Twitch chat. When that happens, you’ll be able to see these principles running live on stream again.
Stay Connected
Want to stay updated on what I’m working on? Here’s where you can find me: