code-reason - LLM Reasoning Benchmark with Code

This is a small experiment for finding the reasoning limit of LLM models. The idea is to generate a function and then let the LLM reason about it. The LLM responds using tool use, and the benchmark calls a verifier that runs the function.

How to Run

This is a node.js TypeScript project.

• Install node.js (tested on v24.16.0).
• Install tsx (tested on v4.22.4)
• Run pnpm i or npm i to install dependencies (I use pnpm but it shouldn't make much difference)
• Run a benchmark. For example:

npx tsx bench.ts --base-url http://localhost:8080 -model gemma4-31b --specs [2] --num-outputs 2 --num-random-inputs 1 --verbose --temperature 0.1 --test forwardPureFunction

Replace base-url with your OpenAI compatible completions URL (this currently uses pi-ai and only supports OpenAI compatible endpoints). I use my local llama.cpp llama-server instance.

Results

Both qwen3.6-27b and gemma4-31b are surprisingly good at the forwardPureFunction test, considering how small they are (although dense). I tested with two branches per level. Qwen had a 100% success rate for functions up to 6 levels deep. This is a 300 line function with 6 inputs. It starts to go down after that. Gemma can do 7 levels with 100% success (~600 lines, 7 inputs). The total token use for this task is around 5,000 tokens for 7 levels, and 10,000 for 8 levels.

So what?

Reflecting a bit about this test, I think a human can do this with functions of arbitrary branching depth. You just have to follow the branches. It's simply not very interesting. It seems that the complexity is linear in the total number of branches. A more interesting test will be exponential in the number of branches, but I haven't been able to think of something systematic yet.

The main goal of this is to compare the reasoning effort required for "understanding" pure functions vs. functions with state. Once I have a worthwhile benchmark, I'll run it also on the proprietary SOTA. It will be interesting to understand what local qwen and gemma provide in comparison in terms of code understanding.

Ideas

• Instead of supplying the inputs, ask the LLM to find inputs the satisfy some condition, such as a specific output. I tried giving a recursive function and ask the LLM to find which inputs result in an infinite loop. For a "simple" test both local models did very well. I don't know how to make such a test with arbitrary complexity yet (A reverse pure function test is already implemented).
• Use local state. I need to think how to do that.
• Use global state. Reason about multiple runs of a function with side-effects.
• Run multiple queries for a single function and see if answers token usage goes down, or some other effect.