=========================================
File 7EVALW98.TXT
Evaluation of results for File, Macro and 
Script Virus/Malware detection under 
Windows-98 in VTC Test "2001-04":
=========================================
Formatted with non-proportional font (Courier)

  Content of this file:
  ************************************************************************
    Eval W98:   Development of detection rates under Windows-98:
  ************************************************************************
    Eval W98.01: Development of W-98 Scanner Detection Rates
    Table W98-A: File/Macro/Script Virus Detection Rate in last 6 VTC tests
    Eval W98.02: Evaluation of overall W-98 AV detection rates (zoo,ITW)
    Eval W98.03: In-The-Wild Detection under W-98
    Eval W98.04: Evaluation of detection by virus classes under W-98
             W98.04.1 Grading the Detection of file viruses under W-98
             W98.04.2 Grading the Detection of macro viruses under W-98
             W98.04.3 Grading the Detection of script viruses under W-98
             W98.04.4 Grading of Poly-virus detection under W-98
             W98.04.5 Grading of VKit virus detection under W-98
    Eval W98.05: Detection of Packed File and Macro Viruses under W-98
    Eval W98.06: Evaluation of False Alarms (File,Macro) under W-98
    Eval W98.07: Detection of File and Macro Malware under W-98 
    Eval W98.08: Detection of "Exotic" malware under W-98
    Eval WNT.SUM Grading of W-98 products
  ************************************************************************

This part of VTC "2001-04" test report evaluates the detailed
results as given in section (files):
    6FW98.TXT       File/Macro Viruses/Malware results    W-98 


The following (24) products participated in this scanner test for W-98 products:
         --------------------------------------------------------
           Products submitted for aVTC test under Windows-98:
         --------------------------------------------------------
         ADO
	 ATR
   	 AV3         	v: 3.0.304.0		sig: Dec.04,2000
    	 AVG 6   	v: 6.220                sig: Dec.11,2000
    	 AVK 10  	v: 10,0,0,0		sig: Dec.07,2000
    	 AVP Platinum   v: 3.5.311.0		sig: Dec.07,2000
         CLE
    	 CMD     	v: 4.60			sig: Dec.11,2000
	 DRW     	v: 4.21		       (see problems list)
         DSE
    	 FPR     	v: 3.08b	        sig: Dec.11,2000
    	 FPW     	v: 3.08b		sig: Dec.11,2000
    	 FSE     	v: 5.21			sig: Dec.01,2000
    	 INO     	v: 4.53 Enterprise Ed.  sig: Dec.11,2000
         MR2
   	 NAV     	v: 5.01.01		sig: Dec.07,2000
    	 NVC     	v: 4.86			sig: Dec.01,2000
    	 PAV     	v: 3.0.132.4		sig: Dec.07,2000
    	 PER     	v: 6.60			sig: Nov.30,2000
    	 PRO     	v: 7.0.A11		sig: Dec.09,2000
	 QHL
    	 RAV     	v: 8.1.001		sig: Dec.11,2000
    	 SCN     	v: 4.12.0		sig: Dec.06,2000
    	 VSP     	v: 12.02.2		sig: Dec.11,2000
         --------------------------------------------------------

Eval W98.01: Development of Scanner Detection Rates under Windows-98:
=====================================================================
The following table summarizes results of file, macro and script virus detection 
under Windows-98 in last 6 VTC tests:

Table W98-A: Comparison: File/Macro/Script Virus Detection Rate  in last 6 VTC tests under W-98: 
================================================================================================
Scan     ------- File Virus Detection -------+---------- Macro Virus Detection ----------+ --ScriptVirusDet--                                                
ner      98/10 99/03 99/09 00/04 01/04 DELTA I 98/10 99/03 99/09 00/04 00/08 01/04 DELTA I 00/08 01/04 DELTA                       
---------------------------------------------+-------------------------------------------+-------------------
ACU        -     -     -     -            -  I    -  97.6     -     -     -     -    -   I   -      -     -
ADO        -     -     -     -    99.9    -  I    -    -      -     -     -   99.9   -   I   -    99.8    -
AN5        -     -   87.2    -      -     -  I    -    -    89.3    -     -     -    -   I   -      -     -
ANT      91.3    -   86.5  92.8     -     -  I  84.3   -    89.5  90.2  96.4    -    -   I  55.2    -     -
ANY        -     -     -     -      -     -  I  70.7   -      -     -     -     -    -   I    -     -     -
ATR        -     -     -     -      -     -  I    -    -      -     -     -     -    -   I    -    2.7    -
AVA/3    96.6  97.6  97.2  97.5   95.2  -2.3 I  96.7  95.9  93.9  94.3  94.1  95.7 +1.6  I  15.0  30.0 +15.0
AVG        -   87.3  87.0  85.4   81.9  -3.3 I    -   82.5  96.6  97.5  97.9  98.3  0.9  I    -   57.9    -
AVK      99.6  90.8  99.8  99.7   99.8  +0.1 I  99.6  99.6 100.0  99.9  100~  100~  0.0  I  91.2  99.8  +8.6
AVP      99.9  99.9  99.8  99.9   99.9   0.0 I 100.0  99.2 100.0  99.9  100~  100~  0.0  I  88.2  99.8 +10.6
AVX        -   74.2  75.7  77.4     -     -  I    -     -   98.7  94.5  99.0    -    -   I  61.4    -     -
CLE        -     -     -     -     0.1    -  I    -     -     -     -     -    0.0   -   I   4.2   6.3  +2.1
CMD        -     -   98.4  99.6   97.8  -1.6 I    -     -   99.6 100.0  100%  100%  0.0  I  93.5  96.9  +3.4
DSS/DSE  99.9  99.9    *   99.8   99.9  +0.1 I 100.0 100.0    *  100.0  100%  99.9 -0.1  I  95.8  100%  +3.2
DRW/DWW    -   89.5  98.3  96.7   98.5  +1.8 I    -   98.3  98.8  98.4    -   98.0   -   I    -   95.6    -
ESA        -     -     -   58.0     -     -  I    -     -     -   88.9    -     -    -   I    -     -     -  
FPR/FMA    -   93.9  99.4  99.7   97.8  -1.9 I  92.4  99.8  99.7 100.0    -   100%   -   I    -   96.9    - 
FPW        -     -   99.2  99.6   97.8  -1.6 I    -     -   99.9 100.0  100%  100%  0.0  I  90.8  96.9  +6.1
FSE      99.8 100.0  99.9 100.0   99.7   0.1 I 100.0 100.0 100.0 100.0  100%  100%  0.0  I  96.7  100%  +3.3
FWN        -     -     -            -     -  I  99.6  99.7  99.9  99.8    -     -    -   I    -     -     -
HMV        -     -     -            -     -  I    -   99.5    -     -     -     -    -   I    -     -     -
IBM      92.8    *     *     *      *     *  I  94.5    *     *     *     -     -    -   I    -     -     -
INO      93.5  98.1  97.1  98.7   97.9  -0.8 I  88.1  99.8   98.1 99.7  99.8  99.7 -0.1  I  78.1  92.7 +14.6
IRS      96.7  97.6    -     -      -     -  I  99.0  99.5    -     -     -     -    -   I    -     -     -
ITM        -   64.2    -     -      -     -  I    -     -     -     -     -     -    -   I    -     -     -
IVB        -     -     -     -      -     -  I  92.8  95.0    -     -     -     -    -   I    -     -     -
MKS        -     -     -     -      -     -  I    -     -     -   97.1    -   44.2   -   I    -     -     -
MR2        -     -   65.9    -    50.1    -  I    -     -   64.9    -     -     -    -   I    -   85.1    -
NAV        -   96.8  97.6  96.8   93.9  -3.1 I  95.3  99.7  98.7  98.0  97.7  97.0 -0.7  I  36.6  65.5 +28.9
NOD        -   97.6  98.3  98.3     -     -  I    -   99.8 100.0  99.4    -     -    -   I    -     -     -
NV5        -     -   99.0    -      -     -  I    -     -   99.6    -     -     -    -   I    -     -     -
NVC      93.6  97.0  99.0  99.1   98.1   0.1 I    -   99.1  99.6  99.9  99.9  99.8 -0.1  I  83.7  88.5  +1.8
PAV      98.4  99.9  99.6 100.0   99.7   0.4 I  99.5  99.5  86.7  99.9  100~  99.5 -0.5  I  90.2  99.8  +9.6
PCC        -   81.2    -     -      -     -  I    -   98.0    -     -     -     -    -   I    -     -     -
PER        -     -     -     -      -     -  I    -     -     -   53.7  67.2  68.5 +1.3  I  18.0  22.0  +4.0
PRO        -   37.3  39.8  44.6   69.9 +25.3 I    -   58.0  61.9  67.4  69.1  67.1 -2.0  I  12.1  40.7 +28.6
QHL        -     -     -     -      -     -  I    -     -     -    0.0    -    0.0   -   I   6.9    -     -
RAV      84.9    -   86.9  86.5   93.6  +7.1 I  92.2    -   98.1  97.9  96.9  99.6 +2.7  I  47.1  84.9 +37.8
SCN      86.6  99.8  99.7 100.0   99.9  -0.1 I  97.7 100.0  99.8 100.0  100%  100%  0.0  I  95.8  100%  +4.2
SWP      98.4    -   99.0  99.6     -     -  I  98.6    -   98.5  98.6    -     -    -   I    -     -     -
TBA      92.6    *     *     *      *     *  I  98.7    *     *     *     *     *    -   I    -     -     -
TSC        -   55.3  53.8    -      -     -  I    -   76.5  64.9    -     -     -    -   I    -     -     -
VBS        -     -     -     -      -     -  I  41.5    -     -     -     -     -    -   I    -     -     -
VBW        -   26.5    -     -      -     -  I  93.4    -     -     -     -     -    -   I    -     -     -
VET        -   66.3    *     *      *     *  I    -   97.6    *     *     *     *    -   I    -     -     -
VSP        -   86.4  79.7  78.1   64.9 -13.2 I    -    0.4   0.3    -     -    0.0   -   I    -   85.3    -
---------------------------------------------+-------------------------------------------+-------------------
Mean     95.0  84.2  89.7  91.6   87.4% +0.4%I  92.1  90.3  93.5  95.0   95.6 84.7% +0.2%I  61.0  76.0 +11.4%
---------------------------------------------+-------------------------------------------+-------------------
Remark: for abbreviations of products (code names), see appendix A5CodNam.txt.

The number of scanners in test has grown to 22. But as some new products reached very low detection
rates, mean detection rates for file and macro virus detection are significantly reduced.

Generally, the ability of W-98 scanners to detect file zoo viruses "in the mean" is decreasing to
an insufficient level. But those products which also participated in the last test (where the mean
detection rate was 91.6) improved their detection rates modestly (+0.4% mean).

Concerning macro viruses, "mean" detection rate is significantly reduced (esp. due to several 
products with very insufficient detection rates), but those products which participated in last 
test on a still acceptable level (95.6%) further improved their detection rates slightly (+0.2%).

Concerning script viruses which is presently the fastest growing sector, the detection rate
is significantly improved though still low (76% mean). But those (16) scanners which participated 
in last test improved their detection rate by impressing 11.4% "in the mean".

   **************************************************************
   Findings W98.1: For W-98, file and macro zoo virus detection rates 
                   are decreasing in the mean. 5 (of 22) scanners 
                   detect almost all zoo file viruses (>99.8%), 
                   whereas 4 (of 22) scanners detect ALL macro zoo 
                   viruses (and 2 more almost all). Detection rates 
                   for script viruses are, in the mean, still 
                   inacceptably low (76%) but the mean detection rate 
                   of those scanners which also participated in last 
                   VTC test is significantly improving, and 3 (of 16) 
                   scanners detect ALL zoo script viruses.
    ********************************************************************


Eval W98.02: In-The-Wild (File,Macro,Script) Detection under W-98
=================================================================
Concerning "In-The-Wild" viruses, the following grid is applied:
       - detection rate is 100% : scanner is "perfect"
       - detection rate is >99% : scanner is "excellent"
       - detection rate is >95% : scanner is "very good"
       - detection rate is >90% : scanner is "good"
       - detection rate is <90% : scanner is "risky"

100% detection of In-the-Wild viruses also esp. detecting ALL
instantiations of those viruses is now ABSOLUTE REQUIREMENT, for
file, macro and script viruses. 

The following 3 W-98 products (of 22) reach 100% for ITW file, macro 
and script virus detection for all objects and are rated "perfect" 
in this category (alphabetically ordered):

                                            ITW Viruses&Files
                                        ( File    Macro   Script)
                                        -------------------------
"Perfect" W-98 ITW scanners:    AVK      (100.0%  100.0%  100.0%) 
				DSE      (100.0%  100.0%  100.0%) 
                                SCN      (100.0%  100.0%  100.0%) 
                                         ------------------------

The following scanners reached the grade "perfect" for file and macro 
ITW virus detection and did also detect all ITW script viruses but not 
in all instantiations (files):

                                            ITW Viruses&Files
                                            ( File    Macro )
                                        ---------------------
"Perfect" file/macro only ITW scanners: ADO  (100.0%  100.0%)
					AVG  (100.0%  100.0%)
                                        AVP  (100.0%  100.0%)
                                        CMD  (100.0%  100.0%)
					DRW  (100.0%  100.0%)
                                	FPR  (100.0%  100.0%) 
                                	FPW  (100.0%  100.0%)
                                	FSE  (100.0%  100.0%) 
                                	INO  (100.0%  100.0%)
					NAV  (100.0%  100.0%)   
                                	NVC  (100.0%  100.0%)
					PAV  (100.0%  100.0%)
				        ---------------------  

   ************************************************************
   Findings W98.2:  3 AV products (out of 22) detect ALL
                    In-The-Wild file, macro and zoo viruses in
	            ALL instantiations (files):
				                 AVK, DSE, SCN
                    12 more products can be rated "perfect" con-
                    cerning detection of file and macro viruses 
                    but they still fail to detect all script viral
                    files (objects): 
                                   ADO, AVG, AVP, CMD, DRW, FPR, 
                                   FPW, FSE, INO, NAV, NVC, PAV		    
   ************************************************************


Eval WNT.03: Evaluation of overall W-98 AV detection rates (zoo,ITW)
====================================================================
The following grid is applied to classify scanners:
     - detection rate     =100% : scanner is graded "perfect"
     - detection rate above 99% : scanner is graded "excellent" 
     - detection rate above 95% : scanner is graded "very good"
     - detection rate above 90% : scanner is graded "good"
     - detection rate of 80-90% : scanner is graded "good enough"
     - detection rate of 70-80% : scanner is graded "not good enough"
     - detection rate of 60-70% : scanner is graded "rather bad"
     - detection rate of 50-60% : scanner is graded "very bad"
     - detection rate below 50% : scanner is graded "useless"

To assess an "overall AV grade" (including file, macro and script virus
virus detection, for unpacked objects), the lowest of the related results
is used to classify each scanner. Only scanners where all tests were 
completed are considered. (For problems in test: see 8problms.txt).

Besides grading products in related categories according to their
performance, it is interesting to compare how products developed. 

The following list indicates those scanners graded into one of 
the upper three categories, with file and macro virus detection 
rates in unpacked samples, and with perfect ITW virus detection 
(rate=100%). 

Under W-98, NO product reached 100% detection rate for file, macro and 
script viruses, both zoo and In-The-Wild, and could be rated "perfect"
(last time, 1 product was rated "perfect"). But 3 scanners are graded
"Excellent" (>99%), and 3 more scanners are rated "very good" (>95%):

                                   (zoo:file/macro/script;file/macro/script:ITW) 
                                   ----------------------------------------------
    "Perfect" W-98 scanners:  NONE
                                   ----------------------------------------------     
    "Excellent" W-98 scanners: AVK      ( 99.8 100% 100% ; 100% 100% 100% )   
                               DSE      ( 99.9 100~ 99.8 ; 100% 100% 100% )  
                               SCN      ( 99.8 100~ 99.8 ; 100% 100% 100% )
			       ADO      ( 99.9 99.9 99.8 ; 100% 100% 100% ) 
                               FSE      ( 99.7 100~ 99.8 ; 100% 100% 100% )  
			       PAV      ( 99.7 99.5 99.8 ; 100% 100% 100% )
                               DSE      ( 99.0 99.9 100% ; 100% 100% 100% )
                                   ----------------------------------------------
    "Very Good" W-98 scanners: CMD      ( 97.8 100% 96.9 ; 100% 100% 100% ) 
			       FPR      ( 97.8 100% 96.9 ; 100% 100% 100% )  
			       FPW      ( 97.8 100% 96.9 ; 100% 100% 100% ) 
                               DRW      ( 98.5 98.0 95.6 ; 100% 100% 100% ) 
   				   ----------------------------------------------

Moreover, 3 more products would be rated "Excellent" for file and macro virus 
detection but with lesser grades for script virus detection:

			       INO      ( 97.9 99.7 92.7 ; 100% 100% 100% )
                               NVC      ( 98.1 99.8 88.5 ; 100% 100% 100% )
    
   ******************************************************************
   Findings W98.3: No W98 product is overall rated "perfect".
                   7 "excellent" overall scanners: 
			AVK, DSE, SCN, ADO, FSE, PAV, DSE
                   4 "very good" overall scanners: CMD, FPR, FPW, DRW
   ******************************************************************


Eval W98.04: Evaluation of detection by virus classes under W-98:
=================================================================
Some scanners are specialised on detecting some class of viruses 
(either in deliberately limiting themselves to one class, esp. 
macro viruses, or in detecting one class significantly better than 
others). It is therefore worth notifying which scanners perform best 
in detecting file, macro and script viruses. 

Two special tests of file viruses were also performed to determine the
quality of AV product maintenance. One test was concerned with almost
11,000 viruses generated from the VKIT virus generator. Some AV products
count each of the potential 14,000 viruses as new variant while others
count all VKIT viruses just as ONE virus. Fortunately, a high proportion
of tested products detects these viruses (see 4.5), although reliability
of detection is significantly less than normally (see 6BDOSFIL.TXT).

Another special test was devoted to the detection of 10,000 polymorphic 
generations each of the following 6 polymorphic viruses: Maltese.Amoeba, 
MTE.Encroacher.B, NATAS, TREMOR, One-Half and Tequila. Detection rates 
were "almost perfect". 

Products rated "perfect" (=100%), "excellent" (>99%) and "very good" 
(>95%) are listed.

    W98.04.1 Grading the Detection of file viruses under W-98:
    ----------------------------------------------------------
    "Perfect" W-98 scanner:         === NONE ===
    "Excellent" W-98 scanners:      ADO  ( 99.0%)
				    AVP  ( 99.9%)      
				    AVK  ( 99.8%)       
				    SCN  ( 99.8%)  
                                    FSE  ( 99.7%)
				    PAV  ( 99.7%)
                                    DSE  ( 99.0%)      
    "Very Good" W-98 scanners:      DRW  ( 98.5%)
				    NVC  ( 98.1%) 
				    INO  ( 97.9%)       
				    CMD  ( 97.8%)      
				    FPR  ( 97.8%)      
                                    FPW  ( 97.8%)     
				    AV3  ( 95.2%)      

     W98.04.2 Grading the Detection of macro viruses under W-98:
    ------------------------------------------------------------
    "Perfect" W-98 macro scanners:  CMD  (100.0%)     
			            FPR  (100.0%)      
				    FPW  (100.0%)      
				    FSE  (100.0%)      
                                    SCN  (100.0%)      
    "Excellent" W-98 macro scanners:AVP  ( 100~ )      
				    AVK  ( 100~ ) 
                                    DSE  ( 99.9%)    
				    NVC  ( 99.8%)     
                                    INO  ( 99.7%)     
				    PAV  ( 99.5%)
                                    ADO  ( 99.0%)      
    "Very Good" W-98 file scanners: AVG  ( 98.3%)
                                    DRW  ( 98.0%)
                                    NAV  ( 97.0%)     


     W98.04.3 Grading the Detection of Script viruses under W-98:
     ------------------------------------------------------------
    "Perfect" WNT script scanners:   DSE (100.0%)
                                     FSE (100.0%)    
                                     SCN (100.0%)   
    "Excellent" WNT script scanners: ADO ( 99.8%)
                                     AVK ( 99.8%)
                                     AVP ( 99.8%)
                                     PAV ( 99.8%)
    "Very Good" WNT script scanners: FPR ( 96.9%)
                                     FPW ( 96.9%)
                                     CMD ( 96.5%)
                                     DRW ( 95.6%)


     W98.04.4 Grading of Poly-virus detection under W-98:
     ----------------------------------------------------
     "Perfect" Poly-detectors which detect all instantiations of all (6)
               polymorphic file viruses always reliable:                  
                         ADO, AVG, AVK, AVP, DRW, FSE, PAV
     "Excellent" Poly-detectors which detect all instantiations of all (6)
               polymorphic file viruses but not always reliably:
                     CMD, DSE, FPR, FPW, INO, NAV, NVC, RAV, SCN    
        

     W98.04.5 Grading of VKit virus detection under W-98:
     ----------------------------------------------------
     "Perfect" VKit detectors which detect all generations of VKit viruses
               always reliably: === NONE ===
     "Excellent" Kit detectors which detect all generations of VKit viruses
               viruses but not always reliably:
                       ADO, AVK, AVP, DSE, FSE, PAV, SCN
    
   ****************************************************************
   Finding W98.4: Performance of W-98 scanners by virus classes:
                  ---------------------------------------------
	       Perfect scanners for file zoo:               =NONE=
               Excellent scanners for file zoo:
                                 ADO, AVP, AVK, SCN, FSE, PAV, DSE
               Perfect scanners for macro zoo:   
                                           CMD, FPR, FPW, FSE, SCN
               Perfect scanners for script zoo:      DSE, FSE, SCN	
               Perfect scanners for polymorphic set:            
                                 ADO, AVG, AVK, AVP, DRW, FSE, SCN
	       Perfect scanners for VKit set:               =NONE=
               Excellent scanners for VKit set:
                                 ADO, AVK, AVP, DSE, FSE, PAV, SCN
   ****************************************************************
   

Eval W98.05: Detection of Packed File and Macro Viruses under W-98
==================================================================
Detection of file and macro viruses within packed objects becomes 
essential for on-access scanning, esp. for incoming email possibly 
loaded with malicious objects. It seems therefore reasonable to test
whether at least ITW viral objects compressed with given popular 
methods are also detected. In addition to those 4 packers used
in previous tests (PKZIP, ARJ, LHA, RAR), the following packers were
added to detection tests: WinRAR and CAB.  

Tests are performed only on In-The-Wild viruses packed once (no 
recursive packing). As last test showed that AV products are rather
far from perfect detection of packed viruses, testbed has essentially
be unchanged to ease comparison and improvement.

        A "perfect" product would detect ALL packed viral samples
                    (100%) file AND macro for all (6) packers:
            ---------------------------------------------
            "Perfect" packed virus detectors: ADO and SCN
            ---------------------------------------------
               
        An  "excellent" product would reach 100% detection
            of packed viral samples (file&macro) for at least 5 packers:
            --------------------------------------------------------
            "Excellent" packed macro virus detector: AVK, AVP, PAV
            --------------------------------------------------------
            Remark: these 3 scanners detect all packed file viruses
                    reliably with all packers, but detect macro viruses
                    reliably only with 5 packers.
 
        A "very good" product would detect viral samples (ITW file&macro)
            for at least 4 packers:
            ------------------------------------------------------
            "Very Good" packed macro virus detector: 
                                      AVG, CMD, DRW, FPR, FPW, NAV
            ------------------------------------------------------
   
Concerning only detection of packed file virus samples, the following products
can rated "perfect" as they detect ALL samples:
	    --------------------------------------------------------------
            "Perfect" packed file virus detectors: ADO, AVK, AVP, PAV, SCN
            --------------------------------------------------------------

Concerning only detection of packed macro virus samples, the only products
rated "perfect" are those (2) which detect ALL (file AND macro) viral samples:
	    ------------------------------------------------
            "Perfect" packed macro virus detectors: ADO, SCN
            ------------------------------------------------

Remark: Much more data were collected on precision and reliability of
virus detection in packed objects. But in the present state, it
seems NOT justified to add differentiation to results discussed here.

   *************************************************************************
   Findings W98.5: Detection of packed viral objects needs improvement
                "Perfect" packed file/macro virus W98 detector:  ADO, SCN
                "Excellent" packed file/macro virus W98 detectors:
                                                               AVK, AVP, PAV
                "Very Good" packed file/macro virus W98 detectors:
						AVG, CMD, DRW, FPR, FPW, NAV
                "Perfect" for packed file virus only:
                                                     ADO, AVK, AVP, PAV, SCN
                "Perfect" for packed macro virus only:              ADO, SCN
   *************************************************************************                        


Eval W98.06: Avoidance of False Alarms (File, Macro) under W-98:
================================================================
First introduced in VTC test "1998-10", a set of clean (and non-malicious) 
objects has been added to the file and macro virus testbeds to determine 
the ability of scanners to avoid False-Positive (FP) alarms. This ability 
is essential for "excellent" and "very good" scanners as there is no automatic 
aid to customers to handle such cases (besides the psychological impact on 
customers work). Therefore, the grid used for grading AV products must be 
significantly more rigid than that one used for detection.

The following grid is applied to classify scanners:
     - False Positive rate = 0.0%: scanner is graded "perfect"
     - False Positive rate < 0.5%: scanner is graded "excellent"
     - False Positive rate < 2.5%: scanner is graded "very good"
     - False Positive rate < 5.0%: scanner is graded "good enough"
     - False Positive rate <10.0%: scanner is graded "rather bad"
     - False Positive rate <20.0%: scanner is graded "very bad"
     - False Positive rate >20.0%: scanner is graded "useless"

Regarding the ability of scanners to avoid FP alarms, 12 (out of 22) products in test
reported NO SINGLE False Positive alarm both in file and macro zoo testbeds and are 
therefore rated "perfect":

	    -----------------------------------------------------------------
            "Perfect" FP avoiding W-98 scanners: AV3, AVG, AVK, AVP, DSE, INO,
                                                 NAV, PRO, RAV, SCN, VSP
            -----------------------------------------------------------------

                 
Several more (totally 20) W-98 scanners gave NO FP alarm EITHER on clean files:
	    -------------------------------------------------------------------
            "Perfect" file FP avoiding W-98 scanners:     ADO, AV3, AVG, AVK, 
                            AVP, CMD, DSE, FPR, FPW, FSE, INO, MR2, NAV, NVC, 
                                                     PAV, PRO, RAV, SCN, VSP
            ----------------------------------------------------------------

   ******************************************************************     
   Findings WNT.6: Avoidance of False-Positive Alarms has signifi-
                   cantly improved for many scanners:
                   FP-avoiding perfect W-98 scanners:  AV3, AVG, AVK, 
                              AVP, DSE, INO, NAV, PRO, RAV, SCN, VSP
   ******************************************************************


Eval W98.07: Detection of File and Macro Malware under W-98 
===========================================================
Since test "1997-07", VTC tests also the ability of AV products to detect non-viral 
malware. An essential argument for this category is that customers are interested to 
be also warned about and protected from non-viral and non-wormy malicious objects such 
as trojans etc, the payload of which may be disastrous to their work (e.g. stealing 
passwords).  

Since VTC test "1999-03", malware detection is a mandatory part of VTC tests, both for 
submitted products and for those downloaded as free evaluation copies.  

A growing number of scanners is indeed able to detect non-viral malware. The following 
grid (admittedly with reduced granularity) is applied to classify detection of file 
and macro malware:

     - detection rate     =100% : scanner is "perfect"
     - detection rate     > 90% : scanner is "excellent"
     - detection rate of 80-90% : scanner is "very good"
     - detection rate of 60-80% : scanner is "good enough"
     - detection rate of  < 60% : scanner is "not good enough"

Presently, there is NO product which "perfectly" detects all (non-viral) file and macro 
malware in VTC testbed but a the number of "excellent" products (>90%) grows, and 2 
products detect ALL macro malware samples:

Concerning file AND Macro malware detection:
	    ---------------------------------------------------------------
            "Perfect" file/macro malware detectors under W-98: === NONE ===
            ---------------------------------------------------------------
            "Excellent" file/macro malware detectors under W-98:
                                            File / Macro 
					FSE (96.1% 100.0%) 
 					AVP (96.9%  99.3%)
                                        AVK (96.1%  99.3%)
					ADO (96.1%  99.0%) 
					PAV (94.6%  99.3%) 
					FPW (94.0%  99.8%) 
					FPR (94.0%  97.3%)                                    
                                        CMD (93.9%  99.8%)
					SCN (90.4%  99.8%)
              ---------------------------------------------------

Concerning only, macro malware detection 1 product is rated "perfect",
and 9 more reach grade "excellent":
	    ---------------------------------------------------
            "Perfect" macro malware detectors under W-98:
                                        FSE (100.0%) 
            ---------------------------------------------------
	    "Excellent" macro malware detectors under W-98:
                                        CMD ( 99.8%)
					FPW ( 99.8%)
                                        SCN ( 99.8%)
					AVK ( 99.3%)
                                        AVP ( 99.3%)
				        PAV ( 99.3%)
                                        ADO ( 99.0%)
                                        NVC ( 99.0%)
					FPR ( 97.3%)
                                        DSE ( 97.3%)
            ---------------------------------------------------


   *******************************************************************
   Findings W98.7: File/Macro Malware detection under W-98 is slowly 
                   improving. NO product is "perfect" but 9 products
                   are rated "excellent" (>90% detection rate):
                           FSE, AVP, AVK, ADO, PAV, FPW, FPR, CMD, SCN
                   ****************************************************
                   Concerning only macro malware detection, 1 products
                   is rated "perfect":                             FSE
		   And concerning macro malware detection only, 10 more
                   products are rated "excellent": 
                       CMD, FPW, AVK, AVP, PAV, ADO, NVC, FPR, DSE, SCN
    *******************************************************************


Eval W98.08: Detection of "Exotic" malware under W-98
=====================================================
With growing exchange of objects which may be activated under other platforms (esp.
including Java, Linux etc), scanners must also detect related malware to warn
customers before activating such malcode. For the first time in VTC tests, a 
selected (small) testbed of viruses active on other platforms - we presently
call these "exotic" viruses" - has been used to determine the detection quality
of contemporary on-demand scanners (engines, signatures).

Even with "reduced" quality requirements, the following products detect exotic
malware on an initially low level of at least 70% (viruses, files):

                        virus/file
		FSE   (92.2%  92.7%)
                AVK   (90.4%  92.0%)
                AVP   (90.4%  92.0%)
                ADO   (85.2%  82.2%)
                PAV   (80.0%  84.7%)
                SCN   (70.4%  80.3%)

   *******************************************************************
   Findings W98.8: Exotic viruses are detected by scanners under W-98
                   only to a lesser degree. The following products
                   detect at least 70%:   FSE, AVK, AVP, ADO, PAV, SCN
   *******************************************************************


Eval WNT.SUM: Grading of W-98 products:
=======================================
Under the scope of VTCs grading system, a "Perfect W-98 AV/AM product" 
would have the following characteristics: 


    Definition (1): A "Perfect AntiVirus (AV) product"
    --------------------------------------------------
            1) Will detect ALL viral samples "In-The-Wild"
                 AND in at least 99% of zoo samples,
                 in ALL categories (file, boot and script-based
                 viruses), with always same high precision 
                 of identification and in every infected sample,
            2) Will detect ALL ITW viral samples in compressed 
                 objects for all (6) popular packers, and
            3) Will NEVER issue a False Positive alarm
                 on any sample which is not viral.
            Remark: detection of "exotic viruses" is
                    presently NOT rated.

    Definition (2): A "Perfect AntiMalware (AM) product"
    ----------------------------------------------------
            1) Will be a "Perfect AntiVirus product", 
               That is: 100% ITW detection
                        AND >99% zoo detection
                        AND high precision of identification
                        AND high precision of detection
                        AND 100% detection of ITW viruses 
                            in compressed objects,
                        AND   0% False-Positive rate,
            2) AND it will also detect essential forms
               of malicious software, at least in unpacked
               forms, reliably at high rates (>90%).
            Remark: detection of "exotic malware" is 
                    presently NOT rated.


    ******************************************************************
    In VTC test "2001-04", we found *** NO perfect W-98 AV product ***
                       and we found *** No perfect W-98 AM product ***
    ******************************************************************

But several products seem to approach our definition on a rather
high level (taking into account the highest value of "perfect" 
defined on 100% level and "Excellent" defined by 99% for virus 
detection, and 90% for malware detection):


    Test category:        "Perfect"              "Excellent"
    ------------------------------------------------------------------
    W98 zoo file test:        ---          ADO,AVP,AVK,SCN,FSE,PAV,DSE       
    W98 zoo macro test: CMD,FPR,FPW,FSE,SCN          ---
    W98 zoo script test:    DSE,FSE,SCN              ---
    W98 zoo Poly test: ADO,AVG,AVK,AVP,DRW,FSE,SCN   ---
    W98 zoo VKit test:         ---         ADO,AVK,AVP,DSE,FSE,PAV,SCN          
    W98 ITW tests:        AVK,DSE,SCN      ADO,AVG,AVP,CMD,DRW,FPR, 
                                           FPW,FSE,INO,NAV,NVC,PAV   
    W98 pack-tests:         ADO,SCN               AVK,AVP,PAV
    W98 FP avoidance:   AV3,AVG,AVK,AVP,DSE,         --- 
                       INO,NAV,PRO,RAV,SCN,VSP
    W98 Malware Test:          ---           FSE,AVP,AVK,ADO,PAV,FPW,
                                                   FPR,CMD,SCN      
    -----------------------------------------------------------------
    
In order to support the race for more customer protection, we 
evaluate the order of performance in this W-98 test with a simple
algorithm, by counting the majority of places (weighing "perfect" 
twice and "excellent" once), for the first places: 

    ************************************************************
    "Perfect" W-98 AntiVirus product: =NONE=       
    "Excellent" W-98 AntiVirus products:
        1st place:                    SCN            (14 points)
        2nd place:                  AVK,FSE          ( 9 points)
        4th place:                  AVP,DSE          ( 8 points)
        6th place:                    ADO            ( 7 points)
        7th place:                    AVG            ( 5 points)
        8th place:                    PAV            ( 4 points)
        9th place:          CMD,DRW,FPR,FPW,INO,NAV  ( 3 points)
       15th place:               AV3,PRO,RAV,VSP     ( 2 points)
      ************************************************************
    "Perfect" W-98 AntiMalware product: =NONE=
    "Excellent" W-98 AntiMalware products:
        1st place:                    SCN            (15 points)
        2nd place:                  AVK,FSE          (10 points)
        4th place:                    AVP            ( 9 points)
        5th place:                    ADO            ( 8 points)
        6th place:                    PAV            ( 5 points)
        7th place:                CMD,FPR,FPW        ( 4 points)
    ************************************************************
