FuzzGuru’s approach to fuzzing

Recently I have seen a lecture by John of Microsoft about their FuzzGuru framework, apparently their approach to fuzzing is through tight integration with code coverage tools, in similar fashion a recently published paper by Microsoft Research, Automated Whitebox Fuzz Testing, shows that this is in fact Microsoft’s approach to fuzzing.
Though this approach seems to provide good results to Microsoft, I am not sure it is a good approach to the majority of people that develop software, as in the security testing phase there is usually little chance that the source code will be available for code coverage testing.

Some would think that binary form code coverage might work as well, I disagree as generic code coverage will make the fuzzer confused as it would not concentrate on the parser part of the program which our fuzzer needs to test.

We’ve been toying with the idea of implementing both source code coverage and binary code coverage in beSTORM but I’m not sure I’m convinced yet that the code coverage approach is beneficial.

  • dre

    the security testing phase at microsoft is their SDL… as it probably should be at most other places.

    source code can usually be made available… what with the decompiling of Java and the CLR… as well as now C/C++ (i.e. HexRays and many earlier efforts such as boomerang). for Java or C, there is already a test harness with fuzz testing available in open-source with jCUTE (smart fuzz). in C# or other .NET, the commercial vendor Compuware SecurityChecker provides fuzz testing capability as well. these allow real code coverage tracking that binary analysis and dynamic analysis cannot provide.

    others are attempting to include some fuzz testing capabilities early in the SDL process, for example Fortify Software with Defender (C#, etc)… and SPI Dynamics with DevInspect and QAInspect (this is more fault-injection and directory traversal / forced browsing against a web application, however). these are in the early stages of development, and many appear to be too similar (i.e. use the same engine) to their fault-injection scanner counterparts.

    from a purist dynamic analysis approach (runtime, memory, or corefile debugging) you’ll want to read chapter 23 of `Fuzzing: Brute Force Vulnerability Discovery’ on “Fuzzer Tracking”, which specifically addresses the topic of code coverage, as well as path coverage. a fuzzer tracker works by first statically profiling the AUT, then recording which instructions are executing in response to the fuzz testing, and finally cross-referencing the instruction data with the profile data.

    what you guys have done is taken a totally different but valid approach to fuzz testing. bestorm is concerned with intelligent fault detection. your monitoring capabilities are more advanced than your competitors, albeit you do not have a Windows target monitoring capability yet. i would be interested to get your monitoring component working under Mac OS X, or with embedded systems. so you are attempting to measure and analyze the presence of a fault, while fuzzer tracking attempts to measure/analyze which tests to send (start to end).

    i think that most security researchers are concerned more with time (fuzzer tracking) than accuracy (intelligent fault detection) when choosing a tool for dynamic analysis.

    binary analysis is unique in that it can provide more information about coverage to a black box test, but how to measure this coverage completeness is another quite unique problem in and of itself. in a static code analysis test, all of the inputs and code execution paths are well-known. i suggest combining fuzz testing with static code analysis over the binary approach whenever possible – however, many still feel that a) the availability of source code is out of reach, and b) binary analysis can and will find bugs that both static code analysis and runtime dynamic analysis will never find. these are very valid points.

  • http://www.BeyondSecurity.com Aviram

    “you do not have a Windows target monitoring capability yet. i would be interested to get your monitoring component working under Mac OS X, or with embedded systems”

    We do have Windows monitoring capabilities and also solutions for Mac and embedded systems. Contact me for more information (my first name at beyondsecurity.com).