Recent searches
Search options
I'm seriously impressed by the quality of the MoE and even 7B models I've run on my 4 year old M1 over the weekend. This is absolutely solid for most of the really useful ways of LLMs I actually use them for day to day.
Already most of these tools allow me to use local models or cheaply hosted versions of them. I can see the total power used in daily inference be about 10 minutes of running 4 cores.
These models and community finetunes are already much more fun than GPT or claude.
1/
Really really curious about where things will be in a few months. I don't think the big players will stay on their pedestal for much longer.
They just can't compete with the myriad of options a community can provide.
2/2
@mnl While I kind of agree, I think most companies will stay with the big LLMs for convenience and legal reasons.
@andrej really curious about that. I'd think running a local model would be much easier from a legal standpoint (and costs).
@andrej also and this is kind of separate, i'm happy with companies staying on the shitty side of that tech and how to use it.
That's not were this technology is empowering.
@mnl Hard agree :D
@andrej and by companies i mean "big companies". I think the added "slice through tedious stuff" will empower much smaller companies to take on the big players without requiring 2 year runways funded by VCs.
@mnl Costs might tip the scale.
@mnl One starting point is Holger Gelhausen with his sideproject KI-INsigths
https://www.ki-insights.com/ @andrej
@mnl Like everything in computing, ‘it depends’. Assembling a GPU cloud is not the business most businesses want to be in.
Core Competency theory would likely suggest that you rent capabilities that are not the ‘secret sauce’ your company adds. So most companies will not build out a competency in local LLMs because it’s a distraction from what they’re offering.
I definitely think local LLMs will be valuable for *many* tools, but at scale you need someone else to maintain it.
@cypherfox what if they just run on your PC? Otherwise, we'll prob see a fair amount of "saas but we'll on-prem if you want" solutions. But I think we'll just run those things on our devices, at least for coding.
@mnl That’s tooling; for developers it’s a reasonable idea, because developers often have beefier machines. So yes, e.g. JetBrains could theoretically use a local model instead of centralized models on systems that meet a certain capability bar.
I don’t _think_ that’s going to impact the existing players much.
SaaS w/optional on-prem is absolutely a great way to go, but again building out an LLM infrastructure that can be deployed on-prem is _hard_. It’s a different scale than running locally.
@cypherfox I'm not sure if I misunderstand (I mostly focus on coding, because I think that's the one domain #LLMs really shine) but two more general scenarios:
- apple ships a "local #LLM framework", as seems likely they will, opening the door for all kind of osx apps to leverage a local LLM
- hosting opensource models opens the door for plenty of SaaS companies to offer their own AI products eating openai's lunch (as phind and perplexity and sourcegraph and plenty others are already doing).
@cypherfox like, github's "moat" is that copilot is a pretty well engineered tool leveraging the speed of the crappy model. No question to me that opensource is going to catch up once the dust settles.
OpenAI's moat is their model that is increasingly annoying by the day, and their horribly frustrating and jank frontend for it. And they spend a shit ton of money for all of their exposure and hype.
But if the models are not the problem anymore, they're just a crappy web frontend.
@mnl I would hazard a guess that OpenAI makes their real money on API charges, _far_ more than the retail web UI.
It’s _really hard_ to build an LLM infrastructure at scale, and for companies that want to do large quantities of work (think automatic legal discovery analysis, large corpus RAG for knowledge-discovery in data, etc.) they want a company like OpenAI to do the work for them.
Large/medium sized companies will pay OpenAI API fees in order to not have to build their own infrastructure.
@cypherfox sure but i was talking about individuals and small shops. I'm fine with big corp paying other big corp lots of money because there's 5 layers of risk-adverse career-oriented managers. But it means opensource, individual developers and small companies are not shackled. Maybe openai's hegemony comes from big contracts, but that's the part I legit don't care about.
@mnl Absolutely! Open source, individuals and motivated small shops are already unshackled and have been for a while.
OpenAI will remain on their pedestal because B2B LLM tooling is a HUGE market and they’re going to make a lot of money from that, but it’s only inertia that keeps smaller/solo players using them.
Well…that and context length. It’s really frustrating that context length is so limited in open-ish models, but that’s a rant for another day.