How fast must your local AI be?
The obvious answer is: my local AI must be as fast and as smart as the one offered by OpenAI (or other commercial vendor).
"I want the best quality, and I want it now!"
Ok, I get that, but let’s focus on the business reality for a sec. Probably, you won’t be able to match the performance of commercial AI providers. And that is totally fine.
"I deserve the state-of-the-art, nothing less!"
I totally get that. Guess what? I got frustrated when my local AI required 5 minutes to ingest the huge codebase before it started work on the requested feature. Those 5 minutes felt like ages to me. I have checked ten times if the freaking thing is even doing something.
Then the requested feature appeared! I asked my AI developer to adjust it to my needs, and it delivered the updated functionality within 10 seconds (since a significant chunk of my app was in the LLM’s context, it could instantly generate the response).
The bottom line is that neither 5 minutes nor 10 seconds is “too long” from a business perspective. Speed matters, but we should not be obsessed with speed alone. What if your AI provider raises their prices 10x or decides to block your access without any valid reason? The convenience of using super-fast, super-smart AI puts your business at risk if that is the only way you interact with AI.
The best way to leverage AI for your business is to diversify your approach:
- Use commercial AI offering when speed and quality truly matter.
- In parallel, experiment with open-source AI hosted on your hardware - give it the same tasks as for the commercial AI and check if/how it handles them.
- Look for non-real-time-critical tasks (there are plenty of those in any company). Local AI is a great fit for those and can deliver real business value essentially for free.
Experiment and fine-tune your local setup - it will reduce your token usage invoice from your commercial provider and make your enterprise more resilient at the same time.
👉 Out of curiosity, how fast must your local AI be?