Tuesday, January 20, 2015

Debugging techniques

In our C.S. classes, we were often shown a picture like this:
The memory is allocated from the bottom of the stack and the top of the heap. Now, students had asked the obvious question of "what happens when they meet?" The obvious answer was that there would be an exception of some kind. I don't know if we ever probed into how that exception worked though. It's pretty clear that it couldn't be a segmentation fault. The memory on either side of that gap belongs to the process, so there's no invalid address being accessed.

Recently I read that the collision is handled by marking a memory page set between the stack and heap areas as a guard page. When the page is accessed, this signals an interrupt to the processor similar to what occurs in a page fault and thus allows the operating system to resume control and handle the overflow by, for instance, killing the process. Guard pages can also be used for debugging a process with unknown behavior that is presumed to access a certain portion of memory in a critical part of its operation, and this technique is valuable for software that subverts debugging with soft breakpoints (which temporarily modify program code) by checksumming code-in-execution.