Can Symbolic Chain‑of‑Thought Boost LLM Logical Reasoning?

Motivation and Background

Achieving human‑level logical reasoning is essential for artificial general intelligence, enabling systems to solve problems, make decisions, and engage in critical thinking. Although large language models (LLMs) have advanced semantic understanding, their logical reasoning remains limited and challenging, requiring strict evidence evaluation and formal deduction.

Recent efforts integrate LLMs with symbolic solvers, but they typically treat LLMs only as translators while the core reasoning relies on external tools, leading to inflexibility, information loss, and reduced interpretability.

Chain‑of‑Thought (CoT) prompting encourages LLMs to articulate intermediate steps, improving reasoning reliability. However, pure CoT struggles with precise logical computation because natural‑language expressions are too abstract for rigorous logic.

SymbCoT Framework Overview

SymbCoT is a fully LLM‑based framework that eliminates external solvers. It consists of four modules:

Translator : Converts premises and questions from natural language into a structured symbolic representation.

Planner : Decomposes the original problem into manageable sub‑problems and creates a step‑by‑step plan linking premises to the query.

Solver : Executes logical inference using symbolic rules (e.g., first‑order logic, Modus Tollens) to derive an answer.

Verifier : Checks each translation and inference step for semantic equivalence and logical correctness, correcting errors when necessary.

The framework integrates symbolic expressions into CoT, adopts a “plan‑then‑solve” architecture, and introduces a retrospective verification mechanism to ensure faithful reasoning.

Experimental Analysis

SymbCoT was evaluated on five logical reasoning datasets using GPT‑3.5‑turbo and GPT‑4. Results show substantial performance gains over traditional CoT and even over methods that employ external tools such as Logic‑LM.

Additional experiments with constraint‑optimization symbolic rules confirm the framework’s robustness across different logical formalisms.

Analysis of reasoning depth reveals that SymbCoT’s advantage grows with problem complexity, indicating superior handling of intricate logical chains.

Robustness tests demonstrate higher success rates for symbolic syntax execution compared to external‑parser approaches.

By combining symbolic and natural‑language representations, SymbCoT mitigates translation errors and enhances the effectiveness of logical inference.

Trustworthiness evaluation defines reasoning instances as “trustworthy”, “untrustworthy”, or “incorrect”. SymbCoT eliminates untrustworthy cases, ensuring that every inference step follows valid logical rules.

Comparisons between GPT‑3.5 and GPT‑4 show that the newer model amplifies SymbCoT’s performance gains, highlighting a synergistic effect.

Error Analysis

Two critical abilities for successful logical reasoning are identified: (1) planning the correct inference path and (2) ensuring each step follows valid logic. SymbCoT strengthens the second ability but relies on the LLM’s inherent planning capability.

Case Studies

Case 1 (CoT failure): A reasoning problem about a Belgian golfer’s ranking leads CoT to an incorrect conclusion due to missing explicit premises. SymbCoT correctly refrains from asserting the missing premise, preserving logical fidelity.

Case 2 (Prover9 failure): An external prover cannot determine whether reading a book makes a person smarter because of hidden assumptions. SymbCoT leverages LLM understanding to capture the implicit premise and correctly infers increased intelligence.

Conclusion

SymbCoT presents a novel, fully LLM‑driven logical reasoning framework that translates natural language into symbolic form, plans, solves, and verifies reasoning steps. Extensive experiments demonstrate superior accuracy, trustworthiness, and robustness compared to existing CoT and tool‑augmented approaches, highlighting the potential of pure LLM‑based symbolic reasoning.