The State of Data in Go
Building AI vs Managing Data
I’ve been re-reading Designing Data-Intensive Applications as I start to work on a new data platform team. It’s not my first platform team (and it won’t be my last), but whenever you’re starting fresh, it helps to revisit the foundations. Martin Kleppmann’s book has been the foundation of engineering any system that processes data since it came out in 2016. Honestly, if you’re in this space and you haven’t read it, you’re working at a disadvantage.
This time through, a line in chapter 3 stopped me in my tracks:
“The limits of my language mean the limits of my world. — Ludwig Wittgenstein, *Tractatus Logico-Philosophicus* (1922)
That quote immediately took my mind to the Go language ecosystem. Please bear with me. That sentence explains perfectly why Go will never be good at AI. Go is limited in ways that make it not a good fit for AI, but I love Go for data systems. So let’s talk about Data, and how data + AI is not the same as data engineering and when to use my favorite programming language.
Why Go Will Never Be Good at AI
This came up recently at a Go meetup I attended. So I thought I would rehash my opinion here.
Here’s the blunt version: Go is not built for mathematics or vector computation. That’s it. I am anxiously waiting to see people accurately compute Partial Differential Equations in Go, but until then Go is not in a good place mathematically for me to recommend it for AI research (if this is enough of an argument for you, skip this section).
Now, someone always jumps in with: “Well, Python isn’t good at computation either!” And they’re right—pure Python isn’t computationally good at math either. But Python doesn’t need to be. Python became the wrapper language around C, C++, and Fortran because academia adopted it. That foothold turned Python into the lingua franca of AI.
The thing is, Go could wrap the same C and C++ tools that Python does. There’s nothing stopping it. But here’s the rub: there’s no community incentive. Python has entire ecosystems—PyData, NumPy, TensorFlow, PyTorch—because academics needed them and VC-backed or large-enterprise companies funded them. Nobody is paying people full time to build AI libraries in Go. A few passionate individuals are maintaining bindings and wrappers, and I admire that work deeply, but it isn’t sustainable.
And let’s not confuse Google’s AI push with community AI tooling. Yes, members of the Go team are building tools that focus include Gemini clients. But Gemini isn’t an open AI tool—it’s a product. That effort will never replace the grassroots, academic-driven innovation that made Python and AI what it is.
So when I say “Go will never be good at AI,” I mean:
It’s not good for training models.
It’s not good for experimentation—calculating F1 scores, evaluating prompts, or iterating on models.
It’s not good at data exploration. Go’s lack of null semantics alone makes it miserable for handling real-world messy data.
You can hack around these limits (there’s even a Go DataFrame library called Gota that invents its own “null”), but you’re always fighting the language.
And unless Go starts showing up in PhD programs, none of that will change.
So What Is Go Good At?
Now let’s pivot! I’ve been writing Go for over seven years, and I love the language. It is my favorite language and I will use it as my hammer to attack every problem that there is. Professionally in my day job, I am a data engineer. And I engineer data in Go. So, why do I choose Go?
Go excels is in data processing.
I'm talking about:
Logs streaming in at high volume.
JSON documents piling up in S3.
User events firing off millions of times per second.
Batch pipelines moving terabytes of raw data into a warehouse.
That’s Go’s home turf. High throughput, low latency, reliable as hell.
I’ve built systems in Go that could chew through messy time series data, reformat it, standardize it, and push it downstream without breaking a sweat. JSON, YAML, Protobuf, Avro1 sometimes the even Parquet—Go eats those for breakfast. If you’re living in event streams or standardized text payloads, Go is a powerhouse.
And let me shout out Go’s database support: the database/sql package is one of my favorite parts of the standard library. The drivers are excellent, and I honestly prefer Go’s raw SQL integration to the abstractions Pandas or Spark layer on top. I would never use an ORM when I need to write high performance transformation queries. When I want to move data, not model it, Go feels like the right tool.
What About Other Languages?
So, I have noticed, that again, the community is not as excited about Go for data engineer. They really like Rust right now. Why does the data world seem to love Rust more than Go for these same use cases? Honestly, I think it’s cultural. Rust feels like a natural evolution for people coming from C++ (the language of TensorFlow), but it’s safer, modernized, but still close to the metal.
In contrast, Go feels like C-like but without memory hiccups and it’s garbage collected—simple, opinionated, batteries included. For many who are accustomed to C++, Rust is the easier leap. And because a lot of data tooling (like Polars) is born from that world, Rust gets the halo effect.
I don’t necessarily think Rust is better for this work. It just had the right cultural moment.
The funny thing here is Java dominated data engineering until ~2020, but it just kinda died, culturally.
Drawing the Lines Clearly
Here’s how I’d divide the work:
Go excels at processing data: ingesting, cleaning, standardizing, and moving it between systems.
SQL (and SQL-inspired tools) excel at analytics engineering: modeling, transformations, exploratory analysis.
Python remains the home for AI experimentation, because academia said so and the ecosystem followed.
And honestly? That division of labor works. It is what I personally recommend for tech stacks and how I think we can make stable data platforms.
Closing Thought
If you ask me whether Go is good for data engineering? I’ll tell you it’s excellent—as long as you’re clear which part of data engineering you mean.
Go is the language that moves data from one world to another. From transactional systems into operational dashboards, from logs into warehouses, from user events into machine learning pipelines.
CSV? Okay, Go stumbles a little there (typing issues, no concept of nulls).