==========================================
File 7EVALLIN.TXT:
------------------------------------------
Evaluation of results for Macro and Script
    Virus/Malware detection under LINUX 
           in VTC Test "2001-10":
==========================================
Formatted with non-proportional font (Courier)

Content of this file:
=====================
  Eval A: Development of detection rates under Linux:
  **********************************************************************
    Eval LIN.01: Development of Linux Scanner Detection Rates
    Table LIN-A: Macro/Script Virus Detection Rate in last 2 VTC tests
    Eval LIN.02: In-The-Wild Detection under Linux
    Eval LIN.03: Evaluation of overall Linux AV detection rates
    Eval LIN.04: Evaluation of detection by virus classes under Linux
            LIN.04.2 Grading the Detection of macro viruses
            LIN.04.6 Grading the Detection of script viruses
    Eval LIN.05: Detection of Packed Macro Viruses under Linux
    Eval LIN.06: Avoidance of False Alarms (Macro) under Linux
    Eval LIN.07: Detection of Macro and Script Malware under Linux 
    Eval LIN.SUM Grading of LINUX products
    **********************************************************************

This part of VTC "2001-10" test report evaluates the detailed
results as given in sections (files):
    6iLINUX.TXT     Macro/Script Viruses/Malware results    LINUX


The following *9* products participated in this scanner test for Linux products:
 	 --------------------------------------------------------
           Products submitted for aVTC test under Linux (SuSe):
	 --------------------------------------------------------
    	 ANT     v(def): 6.8.0.56 	 sig: June 22,2001
    	 AVK     v(def): 3.0 beta 1.1    sig: June 16,2001
    	 AVP     v(def): 3.0 build 136   sig: June 22,2001
    	 CMD     v(def): 4.61.5          sig: June 25,2001
    	 DRW     v(def): 4.25            sig: June 20,2001
    	 FSE     v(def): 4.11 build 3190 sig: June 20,2001
    	 MCV     v(def): unknown         sig: unknown            
    	 RAV     v(def): 8.0.005,
                         scan eng:8.3    sig: June 25,2001
    	 SCN     v(def): 4.14.0
                         scan eng:4.1.40 sig: June 20,2001
         --------------------------------------------------------      
         Remark: in one case, definition and signature dates
                 could not be derived from installed product.
 


Eval LIN.01: Development of Linux Scanner Detection Rates:
====================================================
Linux-based scanners were tested in "2001-10" for the second time,
though this time for macro and script malware detection only. This 
time, 9 scanners were available for tests, 4 more than in last test.


Table Lin-A: Performance of LINUX scanners in Test 2001-04 and 2001-10:
=======================================================================
Scan I  = File Virus = + === Macro Virus ==== + === Script Virus ====
 ner I     Detection   I       Detection      I        Detection    
-----+-----------------+----------------------+----------------------
Test I    0104  Delta  I   0104  0110  Delta  I   0104  0110  Delta  
-----+-----------------+----------------------+----------------------
ANT  I      -     -    I     -   97.1    -    I     -   81.8    -
AVK  I      -     -    I     -   100%    -    I     -   100%    -
AVP  I    99.9    -    I   100~	 100%  +0.0   I   99.8  100%  +0.2
CMD  I    97.8    -    I   100%  100~   0.0   I   96.9  94.2  -2.7
DRW  I      -     -    I     -   99.5    -    I     -   95.4    -
FSE  I    97.1    -    I   100~	 100~  +0.0   I   96.9  92.3  -4.6
MCV  I      -     -    I     -    9.1    -    I     -   27.6    -   
RAV  I    93.5    -    I   99.6  99.5  -0.1   I   84.9  82.5  -2.4
SCN  I    99.7    -    I   100%  100%   0.0   I   99.8  99.8   0.0
-----+-----------------+----------------------+----------------------
Mean :    97.6%   -    I   99.9% 89.5% -0.~%  I   95.7% 86.0% -1.9%
Without MCV:           I        (99.5%)       I        (93.3%)
-----+-----------------+----------------------+----------------------
Remark: for abbreviations of products (code names), 
        see appendix A5CodNam.txt.

While the majority of macro virus detectors works on a rather high level
(the reduced mean detection rate is essentially influenced by one product
which is new in this test), detection rates for script detectors are sig-
nificantly less developped and need further improvement. 

   ******************************************************************
   Findings LIN.01: Different from the first test (where NO product
                    detected all macro and script viruses), now *2*
                    products - AVK and AVP - detect ALL macro and
                    script virus in Zoo test and are rated "perfect".

		    Moreover, 3 (out of 9) products detect ALL zoo
                    macro viruses:                  AVK, AVP and SCN. 

		    For 8 (of 9) Linux scanners, detection of macro
                    viruses is much better (mean detection rate 99.5%)
                    than detection of script viruses (mean detection 
                    rate 93.3%) but the overall picture is negatively
                    influenced by a newly tested product with very low
                    detection rates in both categories.
    *******************************************************************


Eval LIN.02: In-The-Wild (Macro,Script) Detection under LINUX
=============================================================
Concerning "In-The-Wild" viruses, the following grid is applied:
       - detection rate is 100% : scanner is "perfect"
       - detection rate is >99% : scanner is "excellent"
       - detection rate is >95% : scanner is "very good"
       - detection rate is >90% : scanner is "good"
       - detection rate is <90% : scanner is "risky"
								
100% detection of In-the-Wild viruses also esp. detecting ALL
instantiations of those viruses is now ABSOLUTE REQUIREMENT, for
macro and script viruses (it must be observed that detection
and identification is not completely reliable).
								
Presently, 4 scanners are "perfect": 

                                               ITW Viruses&Files
                                         (         Macro   Script)
                                        -------------------------
"Perfect" LINUX ITW scanners:   AVK      (        100.0%  100.0%) 
                                AVP      (        100.0%  100.0%) 
                                DRW      (        100.0%  100.0%)
                                SCN      (        100.0%  100.0%)
                                         ------------------------
"Excellent" LINUX ITW scanners: CMD      (        100.0%   99.1%)
                                         ------------------------

Detection of ITW macro viruses is significantly better developed than
ITW script virus detection as 7 (out of 9) scanners detect viruses at
100%, though not with complete reliability. The following 6 products 
are rated "excellent ITW macro scanners":
                                         AVK, AVP, CMD, DRW, FSE, SCN.

Concerning ITW script virus detection, the following 4 scanners detect
ALL ITW script viruses (though not with complete reliability) and are 
rated "excellent ITW script scanners":
                                                   AVK, AVP, DRW, SCN. 

   ***************************************************************
   Findings LIN.2:  4 AV product detect "perfectly" all ITW
                      macro and script viruses in >99.9% files: 
                                                   AVK,AVP,DRW,SCN
                    1 AV product is "excellent"                CMD
   ***************************************************************
 

Eval LIN.03: Evaluation of overall LINUX AV detection rates (zoo,ITW)
=====================================================================
The following grid is applied to classify scanners:
     - detection rate     =100% : scanner is graded "perfect"
     - detection rate above 99% : scanner is graded "excellent" 
     - detection rate above 95% : scanner is graded "very good"
     - detection rate above 90% : scanner is graded "good"
     - detection rate of 80-90% : scanner is graded "good enough"
     - detection rate of 70-80% : scanner is graded "not good enough"
     - detection rate of 60-70% : scanner is graded "rather bad"
     - detection rate of 50-60% : scanner is graded "very bad"
     - detection rate below 50% : scanner is graded "useless"

To assess an "overall AV grade" (including macro and script virus
virus detection, for unpacked objects, with the additional requirement
of completely reliable identification and detection), the lowest of the 
related results is used to classify each scanner. Only scanners where 
all tests were completed are considered. (For problems: see 8problms.txt).

The following list indicates those scanners graded into one of 
the upper three categories, with macro and script virus detection 
rates in unpacked samples, and with perfect ITW virus detection 
(rate=100%). 
							

                                       (zoo: macro/script; macro/script:ITW) 
                                       -------------------------------------
    "Perfect" LINUX scanners:    AVK          ( 100% 100% ; 100% 100% )  
                                 AVP          ( 100% 100% ; 100% 100% ) 
                                       -------------------------------------     
    "Excellent" LINUX scanners:  SCN          ( 100% 99.8 ; 100% 100% )  
                                       ------------------------------------- 
    "Very Good" LINUX scanners:  DRW          ( 99.5 95.4 ; 100% 100% )  
  				       -------------------------------------

   **************************************************************
   Findings LIN.03: 2 "Perfect" overall scanners:        AVK, AVP
                    1 "Excellent" overall scanner:            SCN
                    1 "Very Good" overall scanner:            DRW
   **************************************************************


Eval LIN.04: Evaluation of detection by virus classes under LINUX:
==================================================================
Some scanners are specialised on detecting some class of viruses 
(either in deliberately limiting themselves to one class, esp. 
macro viruses, or in detecting one class significantly better than 
others). It is therefore worth notifying which scanners perform best 
in detecting zoo macro and script viruses (with ITW detection in ALL files). 

Products rated "perfect" (=100%), "excellent" (>99%) and "very good" 
(>95%) are listed.


     LIN.04.2 Grading the Detection of macro zoo viruses under LINUX:
    -----------------------------------------------------------------
    "Perfect" LINUX macro scanners:   AVK  (100.0%)
                                      AVP  (100.0%)
                                      SCN  (100.0%)
    "Excellent" LINUX macro scanners: CMD  ( 100~ )
                                      FSE  ( 100~ ) 
                                      DRW  ( 99.5%) 
    "Very Good" LINUX macro scanners:    ---


     LIN.04.3 Grading the Detection of Script zoo viruses under LINUX:
     -----------------------------------------------------------------
    "Perfect" LINUX script scanners:  AVK  (100.0%)
                                      AVP  (100.0%)
    "Excellent" LINUX script scanners:SCN  ( 99.8%)                                 
    "Very Good" LINUX script scanners:DRW  ( 95.4%)


   ****************************************************************
   Finding LIN.04: Performance of LINUX scanners by virus classes:
                   -----------------------------------------------
              3 Perfect scanners for macro zoo:         AVK,AVP,SCN
	      3 Excellent scanners  for macro zoo:      CMD,FSE,DRW

              2 Perfect scanners for script zoo:            AVK,AVP
              1 Excellent scanner for script zoo:               SCN
   ****************************************************************
   


Eval LIN.05: Detection of Packed Macro Viruses under LINUX:
===========================================================
Detection of macro viruses within packed objects becomes 
essential for on-access scanning, esp. for incoming email possibly 
loaded with malicious objects. It seems therefore reasonable to test
whether at least ITW viral objects compressed with given popular 
methods are also detected. The following 6 packers were used
in this tests: PKZIP, ARJ, LHA, RAR, WinRAR and CAB. 

Tests are performed only on In-The-Wild viruses packed once (no 
recursive packing). As previous tests showed that AV products are rather
far from perfect detection of packed viruses, testbed has essentially
be unchanged to ease comparison and improvement.

									
        A "perfect" product would detect ALL packed ITW
                    macro viruses (100%) for all (6) packers:
            -------------------------------------------------------
            "Perfect" packed macro virus detectors: AVK,AVP,CMD,SCN
            -------------------------------------------------------
               
        An "excellent" product would detect ALL packed ITW
                    macro viruses (100%) for 5 packers:
            -------------------------------------------------
            "Excellent" packed macro virus detector:      RAV
            -------------------------------------------------
 
        A "very good" product would detect ALL packed ITW
                    macro viruses (100%) for all 4 packers:
            -------------------------------------------------
            "Very Good" packed macro virus detector:      DRW
            -------------------------------------------------
   
Remark: Much more data were collected on precision and reliability of
virus detection in packed objects. But in the present state, it
seems NOT justified to add differentiation to results discussed here.

   ***************************************************************************
   Findings LIN.05: Detection of packed viral objects needs improvement
                Perfect packed ITW macro virus LINUX detector: AVK,AVP,CMD,SCN
 		Excellent packed ITW macro virus LINUX detector:           RAV
   ***************************************************************************                        

								
Eval LIN.06: Avoidance of False Alarms (Macro) under LINUX:
===========================================================
First introduced in VTC test "1998-10", a set of clean (and non-malicious) 
objects has been added to the macro virus testbeds to determine 
the ability of scanners to avoid False-Positive (FP) alarms. This ability 
is essential for "excellent" and "very good" scanners as there is no automatic 
aid to customers to handle such cases (besides the psychological impact on 
customers work). Therefore, the grid used for grading AV products must be 
significantly more rigid than that one used for detection.

The following grid is applied to classify scanners:
     - False Positive rate = 0.0%: scanner is graded "perfect"
     - False Positive rate < 0.5%: scanner is graded "excellent"
     - False Positive rate < 2.5%: scanner is graded "very good"
     - False Positive rate < 5.0%: scanner is graded "good enough"
     - False Positive rate <10.0%: scanner is graded "rather bad"
     - False Positive rate <20.0%: scanner is graded "very bad"
     - False Positive rate >20.0%: scanner is graded "useless"
									
Regarding the ability of scanners to avoid FP alarms, 4 (out of 9) products in 
test reported NO SINGLE False Positive alarm in macro zoo testbeds and are 
therefore rated "perfect":

	--------------------------------------------------------------
        "Perfect" macro-FP avoiding LINUX scanners: ANT, RAV, SCN, MCV
        --------------------------------------------------------------

   ********************************************************************    
   Findings LIN.06: Avoidance of False-Positive Alarms is insufficient 
                    and needs improvement.
                    FP-avoiding perfect LINUX scanners: ANT,RAV,SCN,MCV
   ********************************************************************


Eval LIN.07: Detection of Macro Malware under LINUX 
===================================================
Since test "1997-07", VTC tests also the ability of AV products to detect non-viral 
malware. An essential argument for this category is that customers are interested to 
be also warned about and protected from non-viral and non-wormy malicious objects such 
as trojans etc, the payload of which may be disastrous to their work (e.g. stealing 
passwords).  

Since VTC test "1999-03", malware detection is a mandatory part of VTC tests, both for 
submitted products and for those downloaded as free evaluation copies. In previous
tests, malware detection was tested for file and macro malware; since test "2001-10",
we also tested for FP-avoidance concerning script malware.

As VTC tests were the first to include malware detection tests, we are very glad
to observe that a growing number of scanners is indeed able to detect non-viral 
malware. The following grid (admittedly with reduced granularity) is applied to 
classify detection of file, macro and script malware:

     - detection rate     =100% : scanner is "perfect"
     - detection rate     > 90% : scanner is "excellent"
     - detection rate of 80-90% : scanner is "very good"
     - detection rate of 60-80% : scanner is "good enough"
     - detection rate of  < 60% : scanner is "not good enough"

For the first time, we can report that 2 scanners - AVP and SCN - detect ALL 
specimen of macro and script malware:

Concerning Macro AND Script malware detection under LINUX:    (macro/script)
	    ----------------------------------------------------------------
            "Perfect" macro&script malware detectors:   AVP  (100.0% 100.0%)
							SCN  (100.0% 100.0%)
            "Excellent" macro&script malware detectors: AVK  ( 99.8% 100.0%)
            "Very Good" macro&script malware detectors: RAV  ( 97.7%  81.8%)
            ----------------------------------------------------------------


Concerning macro malware detection only, 3 products are rated "perfect",
and 4 more reach grade "excellent":
            -------------------------------------------------
            "Perfect" macro malware detectors:   AVP (100.0%)
                                                 CMD (100.0%)
                                                 SCN (100.0%)
	    "Excellent" macro malware detectors: AVK ( 99.8%)
                                                 FSE ( 99.5%)
                                                 RAV ( 97.7%)
                                                 DRW ( 90.8%)
            -------------------------------------------------

Concerning script malware detection only, 3 products are rated "perfect",
and 1 more reaches grade "excellent":
            -------------------------------------------------
            "Perfect" script malware detectors:   AVP (100.0%)
                                                  AVK (100.0%)
                                                  SCN (100.0%)
	    "Excellent" script malware detectors: RAV ( 81.8%)
             -------------------------------------------------


   *******************************************************************
   Findings LIN.07: For the first time, 2 LINUX products are "perfect" 
                    in detecting ALL macro&script malware specimen:
                                                              AVP, SCN
                    Moreover, one product is rated "excellent":    AVK
                    And one product is rated "Very Good":          RAV

                    Detection of macro malware is improving, with
                               3 products rated "perfect": AVP,CMD,SCN
                     and 4 products rated "excellent": AVK,FSE,RAV,DRW

                    But detection of script malware is insufficient 
                    and needs improvement, with
                               3 products rated "perfect": AVP,AVK,SCN
                           and 1 product rated "excellent":        RAV
   *******************************************************************



Eval LIN.SUM: Grading of LINUX products:
========================================
Under the scope of VTCs grading system, a "Perfect LINUX AV/AM product" 
would have the following characteristics: 


    Definition (1): A "Perfect AntiVirus (AV) product"
    --------------------------------------------------
            1) Will detect ALL viral samples "In-The-Wild"
                 AND in at least 99.9% of zoo samples,
                 in ALL categories (macro and script-based
                 viruses), with always same high precision 
                 of identification and in every infected sample,
            2) Will detect ALL ITW viral samples in compressed 
                 objects for all (6) popular packers, and
            3) Will NEVER issue a False Positive alarm
                 on any sample which is not viral.

    Definition (2): A "Perfect AntiMalware (AM) product"
    ----------------------------------------------------
            1) Will be a "Perfect AntiVirus product", 
               That is: 100% ITW detection
                        AND >99% zoo detection
                        AND high precision of identification
                        AND high precision of detection
                        AND 100% detection of ITW viruses 
                            in compressed objects,
                        AND   0% False-Positive rate,
            2) AND it will also detect essential forms
               of malicious software, at least in unpacked
               forms, reliably at high rates (>90%).


    *******************************************************************
    In VTC test "2001-10", we found *** NO perfect LINUX AV product ***
                       and we found *** No perfect LINUX AM product ***
    *******************************************************************

But several products seem to approach our definition on a rather
high level (taking into account the highest value of "perfect" 
defined on 100% level and "Excellent" defined by 99% for virus 
detection, and 90% for malware detection):

    Test category:               "Perfect"            "Excellent"
    ---------------------------------------------------------------
    LINUX zoo macro test:       AVK,AVP,SCN           CMD,FSE,DRW 
    LINUX zoo script test:        AVK,AVP                 SCN
    LINUX ITW tests:          AVK,AVP,DRW,SCN             CMD      
    LINUX pack-test:             AVK,CMD,SCN              ---
    LINUX FP avoidance:       ANT,RAV,SCN,MCV             ---
    ----------------------------------------------------------------
    LINUX Macro Malware Test:   AVP,CMD,SCN        AVK,FSE,RAV,DRW
    LINUX Script Malware Test:  AVP,AVK,SCN              RAV
    ----------------------------------------------------------------
    
In order to support the race for more customer protection, we 
evaluate the order of performance in this LINUX test with a simple
algorithm, by counting the majority of places (weighing "perfect" 
twice and "excellent" once), for the first places: 

    ************************************************************
    "Perfect" LINUX AntiVirus product:       =NONE=  (10 points)   
    "Excellent" LINUX AV products:
        1st place:                    SCN            ( 9 points)
        2nd place:                    AVK            ( 8 points)
        3rd place:                    AVP            ( 7 points)
        4th place:                    CMD            ( 4 points)
        5th place:                    DRW            ( 3 points)
        6th place:                MCV,ANT,RAV        ( 2 points)
        9th place:                    FSE            ( 1 point )
    ************************************************************
    "Perfect" LINUX AntiMalware product:     =NONE=  (14 points)
    "Excellent" LINUX AM products:  
        1st place:                    SCN            (13 points)
        2nd place:                    AVP            (11 points)
        3rd place:                    AVK            (10 points)
        4th place:                    CMD            ( 6 points)
        5th place:                    DRW            ( 4 points)
        6th place:                    RAV            ( 3 points)
        7th place:                    FSE            ( 2 points)
    ************************************************************

