Fixing "positive SP value has been found" error in IDA Pro

Posted on December 16, 2016

Tags: rce, assembly

I am not proficient in software reverse engineering, so when I'm doing such stuff in IDA Pro, I mostly poke with Hex-Rays decompiler output. This time, however, I've stumbled on this error:

SOMEADDR: positive sp value has been found

The error description told me to refer to official IDA Pro documentation, but it wasn’t of much help:

The stack pointer at the specified address is higher than the initial stack pointer. Functions behaving so strangely can not be decompiled. If you see that the stack pointer values are incorrect, modify them with the Alt-K (Edit, Functions, Change stack pointer) command in IDA.

So, I selected the prolematic line, hit Alt+K and had no idea what to put in there. After some googling I've found what the error actually means. When decompiling IDA Pro models every instruction with respect to how it modifies ESP/RSP. It starts with value 0 at the beginning of the function, and thus expects it to be 0 before the ret. If IDA fails modelling some instruction, the SP value for it and all its descendants turns to be incorrect.

I haven’t found any solutions to this on the web, so here is what I came up with:

I've fired up a debugger and break at the start of the problematic function
Checked out current SP value.
Executed a function to the middle of it.
Computed the difference between initial SP value and the current one.
Checked if that difference is equal to what IDA Pro computed (using Alt+K on the same instruction).
If the values don’t match, adjust the offset in IDA.
Else perform the same steps for the first or second half of the function.

So, in essence, I was empirically discovering real SP values and adjusting them in IDA Pro using bisection. This turned out pretty fast to do, and allowed me to use F5 on the fixed function.