Fixing "positive SP value has been found" error in IDA Pro

Posted on December 16, 2016
Tags: rce, assembly

I am not proficient in software reverse engineering, so when I'm doing such stuff in IDA Pro, I mostly poke with Hex-Rays decompiler output. This time, however, I've stumbled on this error:

SOMEADDR: positive sp value has been found

The error description told me to refer to official IDA Pro documentation, but it wasn’t of much help:

The stack pointer at the specified address is higher than the initial stack pointer. Functions behaving so strangely can not be decompiled. If you see that the stack pointer values are incorrect, modify them with the Alt-K (Edit, Functions, Change stack pointer) command in IDA.

So, I selected the prolematic line, hit Alt+K and had no idea what to put in there. After some googling I've found what the error actually means. When decompiling IDA Pro models every instruction with respect to how it modifies ESP/RSP. It starts with value 0 at the beginning of the function, and thus expects it to be 0 before the ret. If IDA fails modelling some instruction, the SP value for it and all its descendants turns to be incorrect.

I haven’t found any solutions to this on the web, so here is what I came up with:

So, in essence, I was empirically discovering real SP values and adjusting them in IDA Pro using bisection. This turned out pretty fast to do, and allowed me to use F5 on the fixed function.