First Blood: AI Finally Broke Through ProgramBench 🩸

Recently I wrote that the vaunted AI agents crashed against the ProgramBench benchmark, where they had to recreate a binary from scratch with only execute permissions. Everyone had a proud 0% back then.

Well, the gate has cracked open. The folks released an update: the new GPT-5.5 (xhigh) was the first to solve the starting task — it fully reversed and wrote a working clone of the cmatrix utility.

But the most interesting part of this report is not the solution itself, but how different models approached the task. It's literally a cross-section of how different categories of developers think.

🤡 Claude Opus 4.7
He decided to write in C. He discovered that there are no ncurses.h header files in the Docker. What does Claude do? He doesn't give up. He parses system binaries via ldconfig and nm -D, manually writes a 100+ line curses_decls.h with typedefs, and links all that goodness to the runtime. Absolutely brilliant systems engineering.

And then he fails on 19 tests. Why?
He checked the validity of the entered color via strcmp instead of strcasecmp. Input like GREEN or Red broke the logic. The model spent 178 API calls ($10.74), built a complex setup through the dynamic linker, but screwed up on a case-insensitive string comparison.

🧠 GPT 5.5
The agent checks the Docker, tries to compile a test C file, and sees that there are no headers for ncurses.
Its logic? "Screw it, I'll write it in Python."

Interestingly, the benchmark authors had to remove one test to count the win for Python.
In the original C binary, when entering a huge number, the program crashed due to integer overflow. The variable overflowed, the delay became tiny, and the matrix flew at maximum speed. The benchmark authors considered this a "feature."
📱 The Python version from GPT-5.5 honestly parsed the number (thanks to Python's arbitrary precision integers, which don't care about limits) and honestly went into time.sleep(1e22). The platform predictably threw OverflowError: timestamp out of range.
The bug in C's atoi() was passed off as system behavior, but Python set things straight.

How much time do we give for conquering this benchmark?