Some Thoughts on the Category of Thoughts

I believe that developing the “category of thoughts” should draw upon the larger “cognitive” tradition. In this tradition, category theory, artificial intelligence and congnitive science all play important roles. “Cognitive trinitarianism” has been present in one form or another for many years. Cognition is an idea of greater complexity than computation, and this idea doesn’t seem to have much bearing on engineering design problems.

On theory on LLMs is that these models will develop sufficiently to achieve artificial general intelligence. This “neural” position was taken up strongly by internet-scale companies such as Google and Facebook (and later by OpenAI), and recent developments seem to reinforce this position. The “symbolic” position seems less relevant now. A recent review paper called these “parametric knowledge” and “explicit knowledge” [@pan:2023:large]. It identifies “hybrid models” that use both explicit and parametric representations. These hybrid systems were also called “neuro-symbolic” in earlier AI work. This has a (Wikipedia page)[https://en.wikipedia.org/wiki/Neuro-symbolic_AI]. A 2021 review paper [@hitzler:2021:neuro] identified 5 basic approaches to potential integration, based on a (lecture by Henry Kautz)[https://www.youtube.com/watch?v=_cQITY0SPiw]

[symbolic Neuro symbolic] refers to an approach where input and output are presented in symbolic form, however all the actual processing is neural. This, in his words, is the “standard operating procedure” whenever inputs and outputs are symbolic.
[Symbolic[Neuro]] refers to a neural pattern recognition subroutine within a symbolic problem solver, with examples such as AlphaGo, AlphaZero, and current approaches to self-driving cars.
[Neuro ∪ compile(Symbolic)] refers to an approach where symbolic rules are “compiled” away during training, e.g. like the 2020 work on Deep Learning For Symbolic Mathematics
[Neuro → Symbolic] refers to a cascading from a neural system into a symbolic reasoner, such as in the Neuro-Symbolic Concept-Learner [8].
[Neuro[Symbolic]] refers to the embedding of symbolic reasoning inside a neural engine, where symbolic reasoning is understood as “deliberative, type 2 reasoning” as common, e.g., in business AI, and including an internal model of the system’s state of attention: Concepts are decoded to symbolic entities in an attention schema when attention to these concepts is high. A goal in the attention schema then signals that deliberative reasoning be executed

The CoALA architecture develops a more systematic proposal to integrate large language models with symbolic systems.

One of Kautz’s patterns was a neural pattern recognition within a symbolic problem solver, exemplified by t

I’m interested the broader view of hybrid systems where a “neural” system interacts with a computational “environment”. in the overlap between “explicit knowledge” and computation. There seems to be a strong overlap between graph rewriting in KRR and computation, including symbolic (computer algebra systems, theorem proving) and numerical (floating point implementation). This convergence is represented in systems like Egglog (which combines Prolog and equality staturation) and Gatlab.jl (which implements categorical data structures for use in numerical computing). Egglog points to the common structure of the two problems and demonstrates how combining the approaches can solve problems that were poorly solved using only one approach.

This seems particularly important as LLMs contribute to software engineering. Google’s Alpha* family of tools combine machine learning with an explicit problem representation implemented in an external software tool. Nic has given me several papers from this space that I need to read. But rather than simply taking the software implementation as given, the LLM may be able to rewrite the software in order to improve the implementation of the “external” system.

I’m also interested in the overlap between category theory and rewriting systems. There have been interesting papers on categorical approaches to rewriting (i.e. AlgebraicRewriting.jl), including an example of an approach to planning in “explicit knowledge” systems.