There are few things that I enjoy more than an intelligent public discourse. I whole heartedly thank you for sharing your points of view on this subject. It has been a true pleasure following this thread today.

I’ve spent much of the day on the sofa reading a copy of the Cliff Stoll’s “The Cuckoo’s Egg” and drinking way too much coffee. Basically, thanks for brightening my Ernesto imposed day in the house.

cheers!

ok, fair enough, i was overgeneralizing when i said viruses get discovered eventually… viruses that pose any sort of real problem for the computer user population get discovered eventually… if a virus affects so few computers that it never crosses paths with a virus analyst, or someone who notices it’s effects on their computer, or someone who is able to determine it’s a virus because they employ generic technologies, or someone who just finds it suspicious enough to submit it for good reason or just because it coincidentally happens to be there while they’re having unrelated computer problems then it’s affecting a very small population indeed and doesn’t represent a tangible problem for computer users as a whole…

“It’s not a definitive comparison, because a real-world malicious virus writer could easily be either more or less sophisticated than ISR in generating viruses, but nonetheless was a perfectly valid test. There is no reason that a virus writer *couldn’t* have used an equivalent algorithm.”

there is no reason they couldn’t have gotten assistance from little green men, either… there are many techniques they *could* use, many more than are testable, many more than an anti-virus product can reasonably be made to account for, so singling out one and testing it instead of sticking to what what they actually use is arbitrary and not particularly valid…

“But they could be. If anti-virus products can’t detect new viruses that are algorithmic derivatives of known viruses, then this is extremely worrisome.”

there are a countably infinite number of derivation algorithms, an anti-virus that tested against them all would not halt… perhaps we can keep our expectations about what an anti-virus should be able to do in the realm of the computationally feasible… if the derivation algorithm is known (ie. for a polymorph) then it’s reasonable to expect anti-virus products to detect all the derivatives, otherwise you might as well be criticising them for simply not being able to detect all possible viruses…

“We don’t know for sure, but ISR is a reputable company with some very notable security researchers working for it.”

security researchers are not virus researchers… over and over again people underestimate the level of specialization present (and required) in the anti-virus field…

“If Avi Rubin says he has an algorithm to morph viruses without destroying their viral characteristics, I believe him.”

and if he says that then i’ll call him on his false authority syndrome and is failure to account for the impact decidability plays in such things…

I call this the talking tortoise problem.

From Terry Pratchett’s ‘Small Gods’:
Tortoise: “How many talking tortoises have you met?”
Brutha: “I don’t know.”
Tortoise: “What d’you mean, you don’t know?”
Brutha: “Well, they might all talk. They just might not say anything when I’m there.”

It’s a general security problem. It’s hard for a company to be really certain its network security isn’t compromised; perhaps the attacker was more sophisticated than the network’s detection techniques. Similarly, there could be a good number of viruses that never get discovered because they are sufficiently well-written and have small target populations.

“first and foremost that the viruses it used were not from the real world and therefore a test using them can’t represent real world performance…”

It’s not a definitive comparison, because a real-world malicious virus writer could easily be either more or less sophisticated than ISR in generating viruses, but nonetheless was a perfectly valid test. There is no reason that a virus writer *couldn’t* have used an equivalent algorithm.

“the probability is high that they were generated algorithmically, which is not how variants are created in the wild…”

But they could be. If anti-virus products can’t detect new viruses that are algorithmic derivatives of known viruses, then this is extremely worrisome.

“verifying all 5500 new viruses output were really viruses (just because you start with a virus doesn’t mean your output is still viral) is also a herculean task and there’s no word on how or even if they did this…”

We don’t know for sure, but ISR is a reputable company with some very notable security researchers working for it. If Avi Rubin says he has an algorithm to morph viruses without destroying their viral characteristics, I believe him.

ken, it goes beyond that… no only is there no statistically representative sample to be had in the wild, in the context of virus detection there is no such thing as a statistically representative sample (that’s why testers generally try hard to use as much of the entire population or some significant subset thereof as they can)… without the possibility of a statistically representative sample the selection bias argument falls apart…

“Therefore, viruses that are undiscovered (because, among other things, AV products aren’t good enough) can’t be checked.”

viruses get discovered eventually so if they don’t make it into this quarter’s retrospective test then they’ll make it into next quarter’s or the quarter after that…

“Retrospective testing, therefore, can give a rough relative comparison of products, but cannot establish an absolute value for how good they are.”

nothing can establish an absolute value for how good they are… everything is an approximation, even if we did have statistically representative samples…

“The Consumer Reports test was a controlled experiment, so it didn’t suffer from this sampling problem.”

no, it suffered from worse sampling problems… first and foremost that the viruses it used were not from the real world and therefore a test using them can’t represent real world performance…

second, creating 5500 new viruses is an herculean task even if they were just variants of existing viruses - the probability is high that they were generated algorithmically, which is not how variants are created in the wild… therefore they are even more unrepresentative of real world viruses…

third, verifying all 5500 new viruses output were really viruses (just because you start with a virus doesn’t mean your output is still viral) is also a herculean task and there’s no word on how or even if they did this… without verification for each and every virus sample their testbed cannot be trusted and neither can their results…

best case scenario for their test is that it measures av performance against lab created viruses rather than viruses from the real world… worst case scenario is that it measures av performance against garbage samples…

It’s this: Retrospective testing has an inherent sampling error. You can’t really test against a random sample of all viruses created in the last few months, you can only test against a sample of all viruses *discovered* in the past few months. Therefore, viruses that are undiscovered (because, among other things, AV products aren’t good enough) can’t be checked. Retrospective testing, therefore, can give a rough relative comparison of products, but cannot establish an absolute value for how good they are.

The Consumer Reports test was a controlled experiment, so it didn’t suffer from this sampling problem.

However, the AV companies *still* have a good point regarding ethics. At Black Hat Europe this year I had dinner with an AV researcher from one of the prominent companies, who told me that they had been talking to an external researcher about theories on developing a new kind of malware, but when he actually went ahead and wrote a proof of concept, they had to cut all contact with him.

That makes sense. Like a lot of the security world, AV vendors have an inherent perverted incentive in that they are trying to reduce the number and severity of viruses but their value-add only increases when these things get worse. Therefore, to protect their integrity (or at least the perception of it), they have to take a hard line against all virus creation. The creation of a virus, any virus, even in a lab, has to be harshly condemned. There’s a perceived slippery slope from AV vendor to extortion operation if they soften that line.

So, as it happens, Consumer Reports and ISE are right. Their technique has definite advantages over restrospective testing. But, on the other hand, the AV companies are right too. They have to condemn that technique, they cannot practice it themselves, and there is some risk associated with other companies practicing it (the size of that risk I don’t know).

they aren’t pissed off because of that… they’re pissed off because consumer reports contracted to have viruses made for the test…

it’s not necessary to create new viruses in order to test how well an anti-virus product deals with new viruses - you only need to use a somewhat out of date (by a few months) anti-virus product and the viruses that were discovered since that last update occurred… this is called retrospective testing and it already displays how bad anti-virus products are against new viruses (some have a detection rates that barely make it into the double digits)…

the controversy is over the fact that consumer reports is being part of the problem rather than part of the solution when they make viruses or have others make viruses for them…