AMTSO Inside and Outside

God bless Twitter.

A day or two ago, I was edified by the sight of two journalists asking each other whether AMTSO (the Anti-Malware Testing Standards Organization) had actually achieved anything yet. Though one of them did suggest that the other might ask me. (Didn’t happen.)

Well, it’s always a privilege to see cutting edge investigative journalism in action. I know the word researcher is in my job title, but I normally charge for doing other people’s research. But since you’re obviously both very busy, and as a member of the AMTSO Board of Directors (NB, that’s a volunteer role) I guess I do have some insight here, so let me help you out, guys.

Since the first formal meeting of AMTSO in May 2008, where a whole bunch of testers, vendors, publishers and individuals sat down to discuss how the general standard of testing could be raised, the organization has approved and published a number of guidelines/best practices documents.

To be more specific:

The “Fundamental Principles of Testing” document is a decent attempt at doing what it says on the tin, and provide a baseline definition for what good testing is at an abstract level.

The Guidelines document provide… errrr, guidelines… in a number of areas:

  • Dynamic Testing
  • Sample Validation
  • In the Cloud Testing
  • Network Based Product Testing
  • Whole Product Testing
  • Performance Testing

Another document looks at the pros and cons of creating malware for testing purposes.

The analysis of reviews document provides a basis for the review analysis process which has so far resulted in two review analyses – well, that was a fairly painful gestation process, and in fact, there was a volatile but necessary period in the first year in particular while various procedures, legal requirements and so on were addressed. There are several other papers in process being worked on

A fairly comprehensive links/files repository for testing-related resources was established here and new resources added, from AMTSO members and others.

Unspectacular, and no doubt journalistically uninteresting. But representing a lot of volunteer work by people who already have full time jobs.

You don’t have to agree with every sentence of every document: the point is that these documents didn’t exist before, and they go some way towards meeting the needs of those people who want to know more about testing, whether as a tester, tester’s audience, producer of products under test, or any other interested party. Perhaps most importantly, the idea has started to spread that perhaps testers should be accountable to their customers (those who read their reviews) for the accuracy and fitness for purpose of their tests, just as security vendors are accountable to their own customers.

[Perhaps I’d better clarify that: I’m not saying that tests have to be or can be perfect, any more than products . (You might want 100% security or 100% accuracy, but that isn’t possible.)

You don’t have to like what AMTSO does. But it would be nice if you’d actually make an effort to find out what we do and maybe even consider joining (AMTSO does not only admit vendors and testers) before you moan into extinction an organization that is trying to do something about a serious problem that no-one else is addressing.

Not speaking for AMTSO