=========================
File 5PROTOCO.TXT
AV Product Test Protocol:
=========================
Formatted with non-proportional font (Courier)


This document specifies the test procedures applied to test the 
precision of detection as well as the reliability of detection of 
PC-based boot, file and macro viruses. Moreover, test procedures 
for determining detection of packed viral objects and non-viral
malware are also described. Where relevant, details concerning 
differences against previous VTC tests (esp.2000-04/08) are given.



1) Hardware and System Software used:
-------------------------------------
Test "2000-04" installation differs from last test (1999-09) essentially 
in updated testbeds (which were frozen on October 31, 1999). Concerning
the ability to detect *many* generations of polymorphic file viruses, 
a special testbed with 10,000 generations each of 6 polymorphic viruses 
was included. Again, a special test was devoted to the detection of 
about 11,000 viruses generated with the VKit generator.

Again, the detection of viral code in packed (file and macro) objects
was tested for the set of In-the-Wild viruses, including 6 popular
packers (ZIP, LHA, ARJ, RAR, WinRAR, CAB). Moreover, a set of non-malicious 
objects was used to determine the ability to avoid false-positive warnings, 
and a special (file/macro) malware database was included to determine the 
degree to which trojan horses are detected.

As in test "2001-04", 5 platforms (DOS, W-98, W-NT, W-2000, Linux) were used. 

The virus databases of BOOT, FILE/VKit/Poly and MACRO viruses was held 
on a Win NT 4.0 SP5 server:

    Win-NT Server (1) has the following hardware:
        Pentium 200 MHz, 64 MB RAM, 2 GB hard disk (boot)
                                    2*4,3 GB data/reports, 
                                    2*9,1 GB virus database (mirror)
                   3 network cards: 2*100 MBit/sec, 1*10 MBit/sec
        Protected against electrical faults (USV: APC 420 VA)
        Operating system: Windows NT Server 4.0 SP 6

    Network:       1* 10 MBit/sec BNC for 20 DOS clients
                   1*100 MBit/sec via 2 cascaded switches 
                         for all other clients with 10 MBit/sec cards
                   1*100 MBit/sec via 100 MBit/sec hub for all other clients
                    

Additionally, 25 clients (15 MS-DOS, 4 Win-98, 3 Win-NT, 2 W-2k, 1 Linux) 
were used for the test. DOS-Clients are essentially used to test AV-products
with boot viruses. DOS-Clients work on MS-DOS 6.22. Hard disks are 
only used for the boot process. All W32 client works under English version. 
Win-NT clients work under Windows NT 4.0 Workstation with SP 5, English 
version. All clients are connected to the server using Microsoft NetBEUI.

Generally, clients were flexibly allocated to optimize scanning processes. 

DOS Clients (15)    have the following hardware:
    15* Intel 80486 DX2 50 MHz, 16 MB RAM, 270 MB hard disk, 10 MBit/sec
                    switched to 5 monitors over switchboard
                    software: MS-DOS version 6.22

Win-98 Clients (4)  have the following hardware:
      2*Pentium    133 MHz, 64 MB RAM, 2 GB hard disk,  10 MBit/sec
        Pentium     90 MHz, 32 MB RAM, 1 GB hard disk, 100 MBit/sec
        Pentium-II 350 MHz, 64 MB RAM, 2 GB hard disk, 100 MBit/sec
               
Win-NT Clients (3) have the following hardware:
        Pentium    233 MMX MHz, 64 MB RAM, 2 GB hard disk, 100 MBit/sec
        Pentium-II 233 MHz,     64 MB RAM, 4 GB hard disk, 100 MBit/sec
        Pentium-II 350 MHz,     64 MB RAM, 4 GB hard disk, 100 MBit/sec

Win-2000 Clients (2) have the following hardware:
        Pentium MMX 233 MHz    196 MB RAM
        Pentium III            128 MB RAM

Linux Client (1) has the following hardware:
        Pentium 166 MHz 64 MB RAM, 100 MBit/Sec
        System: Linux (SuSe) Professional 7.0


Specially developed software supporting semi-automatic execution of 
test scans and evaluation of protocols consist of batch programs and 
scripts (PERL and AWK). Some UNIX programs like AWK, GAWK, JOIN etc 
have also been applied.


2) The Databases of File/Boot/Macro viruses:
--------------------------------------------
An overview of entries in the VTC virus databases (status: October 31,
2000) is given in Appendix 3: "A3TSTBED.zip". TESTBED.VTC contains 
the following entries (in ZIPped form):

    1) In-The-Wild Testbeds:
    ------------------------
    ITW-BOOT.VTC    content of ITW boot virus testbed  
    ITW-FILE.VTC    content of ITW file virus testbed
    ITW-MACR.VTC    content of ITW macro virus testbed
    ITW-SCRI.VTC    content of ITW script virus testbed
    PAC-FILE.VTC    content of packed ITW file virus testbed
    PAC-MACR.VTC    content of packed ITW macro virus testbed
    FP-FILE.VTC     content of File virus FalsePositive Testbed
    FP-MACR.VTC     content of Macro virus FalsePositive Testbed

    2) Zoo (=full collection) Testbeds:
    -----------------------------------
    ZOO-BOOT.VTC    content of full boot virus testbed
    ZOO-FILE.VTC    content of full file virus testbed
    ZOO-MACR.VTC    content of full macro virus testbed
    ZOO-POLY.VTC    content of polymorphic file virus testbed
    ZOO-VKIT.VTC    content of VKIT-generated virus testbed
    MAL-FILE.VTC    content of file malware testbed
    MAL-MACR.VTC    content of macro malware testbed (O:)
    EXOTIC.VTC      content of "exotic" viruses&trojans 
                            (platforms: Java, Linux, OS2, ...)


These entries (which also indicate the multiplicity of infected 
objects in the resp. directory) also conform with related entries in 
scanner evaluation protocols.

All file and boot viruses are sorted into their resp. database 
according to diagnostic messages of three "standard" scanners (AVP, 
DSAV, F-Prot). The database of the file viruses consists of four 
parts. If the three scanners identify a virus with the same name, it 
is stored in the first part of the resp. database, named "NORM", 
as those standard scanners reflect an agreed CARO name. All file 
viruses for which no such agreement on their name is obvious, are
stored in a second directory "NONORM" (formerly Not-Yet-CARO). If a 
virus is operating-system specific, it is stored in resp. directories
named Win95 or OS2. The following file extensions are present in 
the file viruses database: EXE, COM, SYS, BAT and CMD.


    Contents of the file virus database:
    ------------------------------------
     20,564 different file viruses (excluding VKit viruses)
    140,703 files infected each with exactly ONE file virus
         20 different file viruses reported "In-The-Wild" (ITW)
        409 files infected with exactly ONE ITW-virus
     60,000 instantiations of
          6 polymorphic viruses, each with 10,000 different 
            generations (size: 1,8 GByte)
     10,706 file viruses generated with VKit virus generator
    104,640 different objects each infected with one VKit virus
            (size: 452 MByte)
         20 ITW file viruses packed packed with 6 packers
            (ZIP, LHA, ARJ, RAR, WinRAR, CAB)
        409 different objects infected with one ITW file virus and
            packed with 1 of 6 packers: ZIP,LHA,ARJ,RAR,WinRAR,CAB
        664 different entries with non-malicious/non-viral objects
              used for false-positive (fp) test
     12,160 malicious/non-viral objects for malware test 


Similar to the file viruses database, all boot viruses are sorted 
into a special directory structure. The boot viruses are not divided 
into different categories. Boot viruses are stored as images of boot 
sectors and processed with SIMBOOT (see 5PROTOCO.TXT).

    Contents of the boot virus database:
    ------------------------------------
        1,311 different boot viruses
        5,723 images representing ONE boot virus each,
           22 different boot viruses found ITW
          300 images representing ONE boot virus, found "In-The-Wild"
        
          
The macro virus database is organised according to the CARO macro 
naming convention. Related testbeds contain macro viruses known at 
end-April 1998 (see VTCs List of Known Macro Viruses). For each macro
virus, different goat documents were stored to test consistent 
identification and reliable detection.

    Contents of the macro virus database:
    -------------------------------------
      6,233 different macro viruses
     19,387 files infected each with exactly ONE macro virus
        147 different macro viruses reported "In-The-Wild"
      1,347 files infected with exactly ONE ITW-virus
         80 ITW macro viruses in 672 infected objects, packed 
            with one of 4 packers (ZIP,LHA,ARJ,RAR)
        329 totally non-malicious/non-viral objects in 26 different
            directories for fp-test

With fast deployment of script (esp. VBS) viruses, a special testbed
for script viruses was developped (the content of which is reflected
in VTCs List of Known Script Viruses).

   Contents of the exotic virus database:
    -------------------------------------
        477 different script viruses
        904 files infected each with exactly ONE script virus
         16 different script viruses reported "In-The-Wild"
        133 files infected with exactly ONE ITW-virus

A new testbeds has been established to determine how AV products
detect viruses on "exotic" platforms such as OS/2, Linux, Java.
 
    Contents of the exotic virus database:
    --------------------------------------
        115 different exotic viruses (different platforms)
        274 files infected each with exactly ONE script virus


2A) Additional File Malware Databases:
--------------------------------------
Concerning non-viral malware, VTC maintains a large collection of 
trojans, virus generators, droppers, worms such as MIRC, as well as
intended and first generation viruses, etc. A subset of these non-
viral file malware  was tested; from the huge database of potential 
file malware, samples were selected to determine the ability of AV 
products to protect customers from these threats. This testbed included:

      12,160 different specimen of file malware 
             in 6,250 different directories.


2B) Additional Macro Malware Database:
--------------------------------------
Concerning non-viral malware, the subset of non-viral macro malware 
tested is well documented (see VTCs "List of Known Macro Malware" 
which summarizes both viral and non-viral macro malware). This 
testbed included:

       403 specimen of macro malware in 627 different directories. 


2C) Additional test for False Positive Detection:
-------------------------------------------------
In order to test the ability of scanners to avoid "false positive"
alarms on non-malicious non-viral objects (files and macros), 2 sets 
of "clean" objects were mixed into the resp. viral databases.


Clean files collected from several CD-ROMs were used for tests:

      664 non-malicious non-viral objects (*.exe, *.com etc) 
        were stored in 27 different directories.

The list of CD-ROMs used for false positive testing is listed in
appendix 3 (A3TSTBED.ZIP).

Concerning testing for false positive alarms on macro viruses, a set
of
      329 non-malicious non-viral objects (*.doc, *.dot, *.xls)
          were stored in 26 different directories.

Remark: concerning copyright of related CD-ROMS, we use selected active
content to help protecting the copyright holder for wrong allegations
concerning false alarms. We never use the code actively but only for
assurance that scanners dont falsely alarm on these samples. 


3) Testing scanners on standard database of file infecting viruses:
-------------------------------------------------------------------
(Text essentially same as in previous tests: cf. 2000-04 & 2000-08).

The viruses are stored in a huge subdirectory tree, the hierarchical
structure of which reflects the CARO virus naming scheme, with the
samples of each virus stored in the leaf directories of the tree. A
virus can be (and usually is) represented by more than one replicant,
although different viruses are not represented by one and the same
number of replicants. All replicants that contain one and the same
virus, are stored in one and the same directory. If two files are in
two different directories, this means that they contain two different
viruses. Each sample in the CARO subset was at least reported by 
three scanners. 

All efforts have been made to ensure that the samples used during the 
test are natural replicants of working viruses: no Germs, Corrupted 
files, or Intended viruses. Nevertheless, it is possible that we have 
made some mistakes in this aspect. If somebody notices any mistakes
of this kind, we shall appreciate being told about them.

If we received arguments from some AV producer that some sample may
be non-viral, we have removed such a sample from the test if we could 
not immediately prove its virality

Each scanner is run on this directory tree, and the resulting report
file is preprocessed. The preprocessing is done with a set of batch
files, some Unix utilities ported to DOS (sort, join, cut, paste,
awk), and a set of awk scripts.

The preprocessed report contains four columns. The first column con-
tains the directory containing viruses. The second contains the 
number of scanned files in the directory. The third contains the 
number of detected files. The fourth contains the information whether
all files are reliable detected (with the same name). For each 
scanner, the report and the preprocessed data are stored in special 
directory.

Not the whole output of the scanner is contained in the third column,
because this output often tends to be too verbose. We have put there
only the distilled information that we have judged important for that
particular scanner. If we have missed some important information, we
shall appreciate being told about it.

Additional remark for Test 1999-03: with linear but fast growth of 
virus numbers, naming became less organised. When this test was 
prepared, less than 25% virus names could be regarded as "CARO 
agreed", esp. as members of the CARO naming committee were overloaded
in their daily fight against new viruses and in helping victims of 
viral events. While VTC testers hope that the chaotic situation of 
virus naming may improve, we have left the second column out of 
this report. Concerning macro viruses, VTC uses its "List of Known
Macro Viruses and Malware" (which is maintained upon consent of
experts working together in CAROs "VMacro list") as naming standards. 


4) Testing scanners on database of boot sector infecting viruses:
-----------------------------------------------------------------
(Text essentially same as in Vesselin Bontchev's test 1994-07)

The boot sector viruses are kept in a similar subdirectory tree, as
files, containing the images of the infected boot sectors. For the
purposes of the test, we used a program, called SimBoot, developed by
Dmitry Gryaznov. This program is still under development and is not
available to the general public, but we will make it available to
those producers of the scanners, who have reasons to suspect that the
program has unfairly interferred with their product and has not
allowed it to be tested properly.

The program takes a file, of which the first 512 bytes are supposed 
to contain the first sector of a boot sector virus. It then emulates 
a blank formatted floppy disk in drive A:, boot sector of which is
replaced by the image in the file. If the file is smaller than 512
bytes, it is padded with zeroes. If the image contains a valid
diskette BPB which indicates a particular diskette size, a diskette
with that particular size is emulated. If a valid BPB is not found,a
360 Kb diskette is emulated. Currently only the first sector of the
boot sector virus is put on the emulated diskette. Program SimBoot
is able to handle complete viruses, consisting of several sectors, 
but this requires that the file image of the virus conforms to a
particular format. We did not have the time to prepare all our boot
sector viruses in this way, although we are considering to do this 
in the future. One major flaw of this approach is that hard disk, 
and respectively MBRs are not emulated. Testing of a virus which
infects only MBRs (e.g., Tequila) but not boot sectors of floppy
disks, is still done by putting an image of the infected MBR on the
boot sector of the simulated diskette. We understand that this is not
very correct - a scanner may refuse to look for a particular virus on
a diskette boot sector, if it knows that this particular virus just
cannot be there. The author of SimBoot is considering to improve it
in the future, in order to make it able to simulate hard disks too.

Once SimBoot creates the simulated infected diskette, it runs the
scanner to be tested, as specified in the configuration file for 
this scanner. (The configuration files are available in the archive
SCRIPTS.ZIP.) The scanner is supposed to scan the diskette (SimBoot
intercepts all INT 13h requests to drive A: and redirects them to
access the simulated diskette), reports its status in the report 
file, and prompt the user to insert the next diskette to be scanned.

SimBoot intercepts the prompt and simulates user input from the
keyboard. Both the prompt and the required user input are specified 
in the configuration file for each scanner. SimBoot is able to handle
scanners that write their prompts directly to the video RAM. It is
also able to handle scanners that poll directly the keyboard when
waiting for user input instead of using the BIOS. SimBoot is even 
able to simulate changing the status of the floppy drive from Closed 
to Open and then again to Closed, in order to handle those scanners 
which poll the DiskChanged line and in order to figure out when the 
user has put a new diskette.

Methodological remark: SIMBOOT is selected as more "realistic" test
methods would be difficult to practice (e.g. tesitng viruses on 
diskettes requires either a permanent formatting/infection/testing 
or a sequential test of many diskettes). But as any simulated method
(even if as well done as SIMBOOT), this method may be unfair to 
scanners which scan for real floppy characteristics. We have been 
informed that McAfee's Scan works in such a way; in this case, the 
real detection rate of such a product can only be assessed using 
some different test method.

The resulting report of each scanner is further preprocessed with a
similar set of batch files, awk and PERL scripts as the report of 
the file virus scanning. 


5) Testing scanners on ITW databases of boot/file infecting viruses: 
--------------------------------------------------------------------
Since VTC test "1999-03", VTC tests of ITW virus detection are based
on the set of samples prepared from "Wildlist Organisation" for its
monthly publication of "The Wildlist"; we wish to thank Wildlist.Org
for their support on this issue. 

When freezing VTC file, boot and macro virus testbeds (for this test: 
November 30, 1998), samples received from Wildlist.org are compared 
with VTC samples and properly sorted into related testbed entries.
Tests run over the whole "zoo" database, while ITW reports are extracted
from the zoo reports of each scanner.
 

6.) Testing scanners on standard database of Macro Viruses:
-----------------------------------------------------------
All AV scanners are tested against two large macro-related 
database. The main datadabse contains all "zoo" and ITW macro 
viruses, both in uncompressed and compressed forms; mixed into this
dfatabase, there are also specific directories contaiining non-viral
macro objects for false-positive detection. The second (smaller)
database contains all non-viral maco malware (trojans, droppers, 
intendeds etc). All malware included in those databases matches 
the contents of the VTC Macro Virus List, which is published  
regularly (previously: monthly, now at the end of each quarter)
For details, see http://agn-www.informatik.uni-hamburg.de/vtc. 

The malware database contains also some file viruses which are being 
created ("dropped") by macro viruses. We decided to test them in the 
context of the macro malware test because they only appear in the 
context of macro malware.

The directory structure of the virus database reflects the CARO 
naming scheme for macro viruses with all samples of one variant 
stored in one subdirectory. Starting from the root directory of the
database, the first level contains directories describing the host 
software (Word, Word97, Excel, Excel97, Lotus123, AmiPro). The second
level contains subdirectories with the names of the families of the 
viruses and the next level hosts subdirectories of all variants of 
that family, in which the viruses can be found. Optionally (only in
malware database), we have another subdirectory called "FILE" which 
contains the file viruses mentioned above.  

The number of samples for each virus varies between one and 78 
samples (for Concept.A), although the average is 2-3 infected objects
each. Our results are split into two sections: "detection of viruses"
and "detection of files", where "detection of viruses" has two sub-
sections: "unreliable detection" and "unreliable identification".

(An index of the malware databases is available in a3tstbed.zip)

After each scanner is run, all report files are preprocessed by those
AWK scripts already mentioned in the desciption of file virus test.


8) Creating the final summary of the results:
---------------------------------------------
(Text essentially same as in previous test: 2000-04 / 2000-08).

The final evaluations for all tests are similar. Only one report of 
file and macro viruses tests is used to get the total number of files 
in the directory. As for boot viruses, the configuration file from 
Simboot is used (if there was no specific need for manual operation). 
Three new files result from these processes. New files contain the 
directory name and the total number of files in this directory. Each 
preprocessed report is joined with the new file. One AWK-scripts 
evaluates the result of the joining.

The results are listed as follows:
   - The number of viruses (+malware) detected: it is not necessary 
     that all examples of the virus are detected.
   - The number of viruses with unreliable (=inconsistent) 
     identification: all variants of a viruses are detected 
     but at least one sample is identified with a different name.
   - The number of viruses with unreliable detection: here, not all 
     samples of a virus are detected but at least one.

The files containing the preprocessed information mentioned above are
huge, although they are reduced to contain essentially the virus names. 
For all tested scanners (latest version), they are included in a 
separate archive (Scan-Res) for anonymous ftp. 

