Binary Obfuscation from the Top Down

Blog Post created by rapid7-admin on Aug 18, 2009

Originally Posted by Sean Taylor



Full disclosure, here: I love binary-related things. I'm an extremely low-level person. If I get any higher level than C, chances are I'm just being lazy. Otherwise, I'm completely immersed in the world of K&R. So any opportunity I get to peddle my wares in the world of binary I will gladly take. 

What started this whole talk was mdl.cpp-- my extremely tame, proof-of-concept malware. Bored one day, I thought it would be fun if I could avoid anti-virus heuristics in some way or another. So I wrote this mild trojan-horse (whose entire existence was intended as an upload-exec binary), then sent it off to
VirusTotal. Naturally, the heuristics were triggered in various malware scanners as "generic downloader"-- which is exactly what it was. I then proceeded to try just a little bit of information hiding: procedurally yank the functions from the operating system at run-time, then execute them. Not a peep from VirusTotal. I got a thumbs-up of no vendors flagging it as malware. This inspired me to see what else I could do from high-level programming to do the following: avoid heuristics, frustrate reverse-engineering and-- just generally-- do cool, fun and mindblowing things. 

What's interesting is that I'm sure I've only scratched the surface. Levereging the years and years of scientific research behind compiler optimizations against specifically crafted code isn't exactly the simplest of tasks-- "The Dragon Book" is somewhat dry. There's still some clever tricks to be done with regards to high-level binary obfuscation. 

A word of caution, though, if you plan on peeking at the numerous examples and binaries in my talk. "manifest.exe" triggers as a false positive for a generic trojan by a few anti-virus vendors. This is because its bizarre "packing" method triggers some heuristics. (Why some vendors see packing as "bad" automatically is silly, but...) 

So, if you missed my presentation at DEFCON this year, feel free to trawl through the PDF and source-code examples.


Binary Obfuscation from the Top Down PDF
Sourcecode and binary examples