I’m no geography expert, but I didn’t think there were beaches in Atlanta. After reading Dan Holden’s post on ISS’ “Frequency X” blog, I am beginning to doubt this presumed truth. There MUST be beaches in Atlanta… I don’t see any other way that Holden and ISS could have their heads so deep in the sand.

I stumbled upon Holden’s rant, titled “Who do you trust?” in my first (and last) visit to “Frequency X”. Holden talks about a recent ISS advisory and the QuickTime for Java bugs before (in essence) calling TippingPoint a bunch of posers who sublet security research instead of building their own talent.

Disclosure: I used to work for TippingPoint; I no longer do. Accordingly, my views don’t speak for anyone — TippingPoint or my current employer.

Holden writes:

“Well in your humble narrator’s opinion this is what we call ‘rented security’. A customer purchasing their security guidance and protection technology from a company that blindly leases security research from third parties rather than staffing a comprehensive holistic security research staff is akin to buying a fake Rolex. In the end the watch may appear real, you may fool some casual admirers, but it won’t keep accurate time, hold-up, or fool an expert.”

Depending on your definition of “blindly”, TippingPoint’s ZDI program might meet the definition of “rented security”. But it seems Holden’s entire thesis depends upon glossing over a best-of-breed research team. TippingPoint researchers present at the same conferences where ISS employees are sued and then deprived of credit for trying to present… all while customers received no protection for the vulnerabilities that ISS’ own researcher found. I could keep going for another whole paragraph. The “fake Rolex” analogy is miles off, largely because Holden again chooses to ignore TippingPoint’s substantial in-house research. He makes it sound like there’s no talent managing the ZDI, and TippingPoint simply regurgitates purchased information with no review; that claim is obviously baseless.
Holden continues:

“In other words, outsourcing what is supposed to be your core competency as a security organization isn’t a smart business practice. Even Gartner has spoken out about the dangers of this type of activity calling it a ‘risky endeavor’. The real problem is that there is no guarantee whether the information being purchased hasn’t already been shared or really how good or thorough it is.”

Ooh… a “risky endeavor”, according to… Gartner? Now I’m really shaking in my boots. Okay, so maybe I’m not. Maybe I can understand, like most people, that the risk TippingPoint is taking is intended to supplement its research team’s work. Granted, the ZDI is a beast of a supplement, with 92 published advisories in under two years, but it’s still a supplement to a research team that holds its own pretty well. More from Holden…

“However, what happens when you are on a time crunch to bring the cash home before anyone else? Well it’s possible you could miss something which is exactly what happened. While X-Force was in the process of finding and fully exploring the QuickTime bug we found another and immediately notified Apple to responsibly disclose the vulnerability. The patch for this was released yesterday at 4PM EST.

Apple QuickTime Code Execution

Now is it important that we had protection out 3 weeks prior? Well that sure was nice for our customer base but what is truly important is that we understood the original bug and being a true research group that does this type of thing everyday ended up uncovering a related vulnerability in the process.”

What Holden forgets is that “true research groups” make mistakes in assessing vulnerabilities, too. ISS made a huge error of its own with the Apache Chunked Encoding Vulnerability. X-Force Advisory 120 still to this day states (in spite of the Scalper worm, which proves it wrong):

“X-Force has verified that this issue is exploitable on Apache for Windows (Win32) version 1.3.24. Apache 1.x for Unix contains the same source code, but X-Force believes that successful exploitation on most Unix platforms is unlikely.”

ISS “understood the original bug” as well as a fly in your soup on this particular occasion; the timing of their QuickTime find seems like luck. Variants can and do happen (often as a result of shoddy patches, rather than misses by researchers), and that’s true whether you’re contracting research or working commercially. Being a commercial researcher does not intrinsically mean you produce better research. More from Holden…

“…the commercial space isn’t where the most vulnerabilities are found so what is the most important part of vulnerability discovery? Understanding the nuances and severity of the issue and then responsibly working with the vendor to get it rectified. Now this is certainly a complicated subject and some of you may disagree with me.”

If you’re an IPS vendor and I’m buying from you, the most important part of vulnerability discovery is how well your protection adapts to that discovery. Key to adapting to discoveries in a market that is still largely reactive is getting as much raw threat data as early as possible. ZDI does exactly that for TippingPoint, so I of course, disagree. ZDI is a smart, customer-beneficial approach to acquiring as much of the “big picture” of vulnerability intelligence as possible.
Finally, Holden concludes his rampage against purchased research:

“However, until there is a Consumer Reports for the security space as a whole who are you going to trust? Anonymous researcher, or the X-Force guys that wrote the books that anonymous researcher is reading to get his paycheck from who knows what sources?”

Why don’t you tell me… who should I trust? Should I trust X-Force because they’re a good research team and wrote some good books, or should I trust the research team that uses its capital and actively acquires threat information from the outside to complement internally-produced information?

I love how Holden pulls off the “we’re leeter than they are” argument, implying that X-Force authors wrote the books TippingPoint’s ZDI contributors are using. That might have been true five years ago, but it’s hardly the case now. Even if it were, I’d buy from the company that’s producing the raw information. Who cares if ISS team members wrote the books that today’s researchers are using to find bugs? It doesn’t matter to anyone buying today’s protection. The obvious fact is that the research isn’t coming out of ISS anymore, and Holden is trying to stoke the embers of a tired market perception that X-Force is still a top-tier research team.

I digress. Holden’s question was who we should trust. Until ISS establishes a precedent for competing on facts, rather than false spin, at least part of the answer is self-evident: you shouldn’t trust ISS.