Consumer Reports writes viruses

OK, once more. (We, in the AV research field, have been through this endlessly, particularly with Doren Rosenthal back a decade or so.)

Finding “fake” viruses doesn’t prove anything. (How do we know what CR created were really viral? ISE’s involvement is promising, but not definitive. And CR has never shown any particular aptitude for technology, although I always read up on their reports when I’m buying a new blender.)

Creating “new” viruses is a no-no in the AV research field. Yes, it’s a knee jerk reaction going back to the time that the creation of any new virus was a cause for alarm, but nobody has ever any benefit from writing viruses. Ever. (Yes, I’ve heard all the arguments about teaching people to write viruses so they know how
to defend against them, and even using viruses to fight viruses. Nobody has ever been able to demonstrate any benefit. Ever.)

Using “new” viruses to test AVs just means that you guessed closer to the winner’s heuristic algorithm than the others in the test. Doesn’t prove anything about the quality of the programs either way.

What it does prove is that CR was lazy about doing the testing. There are lots of ways to test AVs, but they all involve hard work. (Believe me: I did it for years, and I’ve published more tests of AVs than anyone else.)

For example, CR tells us a little bit about how they tested the stuff.

“We hadn’t seen any independent evaluation of antivirus software that measured
how well products battle both known and new viruses, so we set out to fill that

Obviously they didn’t search very hard: my stuff is old, but it is still online. Also, testing is covered in detail in both RSGCV and VR. And Robert Vibert even did a self-published manual on it a few years back.

(Some references to the difficulty:
The latter was actually written by Alan Solomon: Sarah Tanner was a secretary at VNI or VB, I believe.)

In their explanation of how AV software works, they betray the fact that they haven’t done any research. There are, and always have been, only three basic AV detection measures: signature scanning, change detection, and activity monitoring. They only talk about signature scanning. (The comment that “[d]epending on the manufacturer, that process may take a few days” is interesting: even Fortinet,
which does AV as a sideline in its firewalls, was looking at a 1 hour turnaround three years ago.) The mention of heuristics also betrays a lack of awareness:
heuristic scanning is a static form of activity monitoring, and it isn’t the only “generic” AV (although it is currently the most popular form).

“To see how quickly software makers update their signature lists, we gave all of the products Internet access. Then we spent weeks closely monitoring each product and noted how early, if at all, the manufacturer equipped it to detect newly discovered viruses.”

This is interesting. It’s too bad they don’t go into more detail: it is the type of info that could, in fact, be useful.

“To pit the software against novel threats not identified on signature lists, we created 5,500 new virus variants”

This, of course, is what everyone is talking about. Creating one or two carefully crafted viruses, aimed at threats which might be emerging but not widely used yet, might be interesting. Mass creation of 5,500 viruses means you do it in an automated fashion. Which means you use the same type of heuristic that the AV vendors do when creating their heuristic sigs. So, as noted above, whoever wins used the same heuristic you did, or you used the same one they did. Has nothing to do with what vxers do.

“Then we infected our lab computer with each of 185 of them to see whether the products could better detect viruses that were actively executing, based on their behavior.”

Weird. Need much more detail on this. Did CR infect its lab with real viruses? Or only the artificially created ones?

“Finally, to see how often the antivirus software raised false alarms by identifying benign files as viral, we scanned more than 100,000 clean files.”

Good. Did they also try to disinfect infected files? Did they try to disinfect non-infected files?

From the MSN report:

“it took a handful of existing viruses and created hundreds of slight variants, changing the malicious programs just enough to evade detection by an antivirus program with a list of known threats.”

So you deliberately create the viruses based on how well they evade detection, and then use them to test AVs? Sure, that sounds like a good idea. And how well did the AV do that you were initially using to test whether or not they evaded detection? Where did you get your ideas on how to make new viruses that evade detection, old copies of Phrack? In fact, minor new variants are even worth testing. The real dangers are the completely new viruses, like a Melissa, LoveBug, or Blaster, using a completely new vector or function.

And, yes, if you do go to the page for the actual tests, all you get is a page asking you to subscribe.

Yup, sounds like a PR gimmick to me …

  • David Harley

    Darn. Rob got in first. :) Rather than flood the blog pages with more AV stuff, I’ll just say that at least three major lessons can be learnt from this test.
    1) You don’t -have- to write viruses to test AV heuristics. Reactive testing isn’t perfect, but it sidesteps the ethical issues and most of the technical difficulties. Actually, the worst problems about reactive testing relate to keeping the playing field level, and that applies to -all- comparative testing.
    2) The rest of the security industry still doesn’t understand AV technology, practice or issues: see the comments in SANS Newsbytes.
    3) The AV industry and research community remain -hugely- (and, largely, unfairly) mistrusted.

  • Pingback: Worm Blaster Port Virus Blaster