I don’t love the term sovereign AI, but that seems what we’ve landed on. I think a lot of the open weight models (not just the ones from China, but they are particularly impressive) aren’t being used enough.
OpenAI and Anthropic as risks
There are lots of things we can say about these two giants. While I think a lot of the concern is warranted, right now I want to focus on operational and business risk. Model updates and deprecation can have a material impact on your workflows - not just by adding and removing models, but also smaller tweaks to the system prompt and harness can materially impact how well your workflows run (if at all).
To keep all this stable, you need to own your AI as much as possible. There are three major routes.
- There is the literal owning of the hardware (a GB10 box is ~$5k, and a larger machine which can run bigger models or more models in parallel can be up to $100k)
- There is renting the hardware/inference via AWS or another service
- There is accessing the open models via API
Each option has its own set of tradeoffs, and I think the right option depends on your workflow scenario.
The first options is appealing because it brings everything in house, maximum control but also largest capital outlay. You can run a lot of smaller edge models on your local machine if you have a dev spec one. I do a lot of prototyping on my M1 Max with 64 GB of RAM. It’s 5 years old and still chugging along fine. Making the jump to the DGX or a different bit of hardware will be something that opens up workflow persistence. You can stack those machines fairly easily and I’ve been impressed with their performance. Once you have stable workflows, this is probably an end state you want to look at seriously.
The second option is where I think a lot of people will land. It gives you a good blend of benefits and cost management. You do need to manage your infra, but that can also be offloaded to a service (hi!). You can scale up and down as needed. The case study about building a knowledge system from multiple streams of inputs uses an AWS on-demand machine for heavy processing. You get maximum control and you can run experiments and new models.
The third option actually has the same risk as I lined out above. You have a bit more control (some providers will allow you to tweak the system prompt this way too!) and it’s ideal for quick experimentation. For frontier model needs, and one-off flows, it’s ideal because the cost is hard to beat. Just be aware the risk of an API changing is no different.
Why you want your own stable of models
I’ll expand on this in another post, but having access to AWS (or GCP or Azure) infrastructure to work with open weight models means you can make models tailored to your workflows. For those that you run a lot, a few percent improvement can really move the needle. And since this kind of work (again, hi!) can be done fairly quickly and cheaply, it’s worth experimenting.