thinking step-by-step requires 100% accuracy in each step. If you are 95% accurate in each step, after the 10th step, the accuracy of the reasoning chain drops to 59%. this is the fundamental problem with llm for reasoning.
reasoning requires deterministic symbolic manipulation for accuracy. only then it can be composed into long chains.
Tongue in cheek but this has been considered and has resulted in experiments like tree of thought and various check your work and testing approaches. Thinking step by step is really just another way of saying make a plan or use an algorithm and when humans do either they need to periodically re-evaluate what they’ve done so far and ensure it’s correct.
The trick is training the model to do this as a matter of course and to learn which tool to apply at the right time which is what the paper is about wrt interspersed thoughts.
>reasoning requires deterministic symbolic manipulation for accuracy
No, that is automation. Automated reasoning is a thing, indeed. And I can kind of see a world where there is a system which uses LLM for creative thinking, augmented with automated reasoning systems (think datalog, egg, SMT-solver, probabilistic model checking etc).
all human knowledge is created by a small number of people. most of us just regurgitate and use it.
think euclid, galileo, newton, maxwell, etc...
and all human knowledge is mathematical in nature (galileo said this).
what is meant here is that, facts and events in the world we perceive can be compressed into small models which are mathematical in nature and allow a deductive method.
human genius comprises of coming up with these models. This process is described by Peirce (and Kant before him) ie, inventing concepts and relations between them to comprise models of the world we live in.
imagine compressing all observed motion into a few equations of physics. or compress all electromagnetic phenomena into a few equations. and then use this machinery to make things happen.
imagine if we feed a lot of perceived motion data into a giant black-box (which could be a neural net) - and out comes a small model of that data comprising newton's equations (and similarly maxwellian equations).
But, this giant knowledge edifice is built on solid foundations of mathematical reasoning (newton said this).
human genius is to invent a mathematical language to describe imaginary worlds precisely, and then a scientific method to apply that language to model the real world.
reasoning requires deterministic symbolic manipulation for accuracy. only then it can be composed into long chains.