Static string signatures in Antivirus don't effectively fingerprint unknown malware variants. One approach which has seen some success is using the structural information of a program's control flow to build a signature. The control flow describes the possible flow of execution a program may take. It is represented by what's known as a directed graph - basically a network diagram of how execution moves from one set of instructions to another. Control flow doesn't change much in variants even if the byte level content changes like in polymorphic and metamorphic malware. A real advantage of using graphs is that we can compare these graphs to show if they are approximately similar. We can quantify how similar two programs are and set a threshold to identify related or mutated malware. I have implemented a system using these ideas to perform malware detection in real-time.

The system improves previous work by performing more efficiently and detecting more variants. It replaces the classification system that I presented at Ruxcon 2010 and uses several new ideas that make it better. This presentation discusses how the system works, its implementation, and its evaluation.