Recent searches
Search options
This seems like a big deal.
Meta AI proposes "Large Concept Models" to complement "Large Language Models".
(i.e., reasoning in an embedding space of concepts rather than words)
https://github.com/facebookresearch/large_concept_model
https://arxiv.org/abs/2412.08821
popularized article: https://www.marktechpost.com/2024/12/15/meta-ai-proposes-large-concept-models-lcms-a-semantic-leap-beyond-token-based-language-modeling/
@peterkaminski "LCMs employ a hierarchical structure, mirroring human reasoning processes"
Not a good sign if they think our reasoning is hierarchical.
"reduces sequence length compared to token-level processing, addressing the quadratic complexity of standard Transformers and enabling more efficient handling of long contexts"
Now seeing "concepts" as a kind of compression (of strings of tokens), which I've seen articulated before as a way of understanding much of what's happening with LLMs.
@peterkaminski (There are hierarchies, among our ways of thinking, sure. But ultimately they are not clean or coherent hierarchies, that is they branch in all kinds of weird ways that defy exclusively top-down control, organization, or even understanding.)
@slowenough @peterkaminski and that resistance to coherence and understanding is very good.