===================================
File 8PROBLMS.TXT:
-----------------------------------
List of problems experienced during 
     VTC test "2001-10":
===================================
Formatted with non-proportional font (Courier)


Content of this file:
=====================
    1. Introduction: General Problems 
       1.1 Problems likely related to FindFirst/FindNext anomaly
           (essentially unchanged since last test)
    2. List of benevolently behaving AV products in test "2001-10"
    3. Problems of AV products observed during test "2001-10"
       3.1 List of Postscans
       3.2 List of specific problems


1. Introduction: General Problems:
==================================
For automatic tests on large viral databases, and for automatic 
processing of large scanner log files, a set of test conditions 
is prerequisite for scanners to participate in a VTC test (see: 
4TESTCON.TXT).

In many cases, serious problems were observed during some tests. 
DOS scanners were either not suitably running under SIMBOOT and 
crashed, or problems appeared with the (rather large) file virus 
database. In some cases, scanners crashed upon detecting some 
specific virus; in few cases, "manual" operation instead of 
automatic (batch) operation helped solving some of these problems. 
Such curative action was also applied when possible in cases where 
log files were inadequate (e.g.needing manual operation for export).

With growing velocity of processors, DOS scanners (running without 
any problem on INTEL 386 and 486) growingly crash on Pentium II/III
systems faster than 250 MHz. Another general problem with DOS scanners 
is related to counters for files and viruses which often seem to be 
designed as integers, so they start after 65,536 with 0.

During preparation and test, we again experienced a serious problem 
reported also in previous VTC Tests, according to which management of
large sets of directories in FAT and NTFS may not reliably work. Both
when attempting to move large parts of our file virus database, as 
when some scanner proceeded scanning subsequent viral directories, 
we found that several directories were not moved or touched. This
effect seems to happen stochastically, such that subsequent attempts
gave different results. Concerning omitted (=unscanned) directories,
we overcame this "dysfunctional" behaviour of FAT and NTFS by repeat-
ing scanning so long until the number of scanned files agreed with 
the (known) number of directories in testbeds. Overcoming this  
problem was extremely time-consuming, and this was a reason for 
delaying publication of results. 

In cases where scanners crashed during detection test upon the rather
large file virus database, tests were performed in several runs on
partitions (essentially on directories with same first letters of
names). In most cases (apart those reported below), these tests
were completed, and resulting files were joined and evaluated.

Finally, with growing testbeds, test protocols produced by scanners
grow equally. When processing such protocols, we need meanwhile up
to 6 GByte of disk space, and our evaluation scripts (in AWK) 
become more complex. Under these conditions, we also suffered from
an evident bug in the AWK processor which inhibited proper evaluation
and required additional quality assurance (including time and efforts). 


1.1 Problems very likely related to FindFirst/FindNext anomaly:
---------------------------------------------------------------
In several cases, scanners finished a first scan although they
had not touched all directories with infected objects. In such
a case, a postscan was started adressing only those untouched
objects; a second postscan was started when again objects were
observed untouched, but after the 2nd postscan, no more scan
was started. This behaviour may originate from a reported anomaly
in the behaviour of FindFirst/FindNext (those routines are used
to handle objects in directory trees) which has not been cured
so far by Microsoft.

In the "problems list", these postscans are ked as "minor problems":
different from crashes, postscans "only" required time for running
the tests and evaluating test protocols (which significantly delayed
results).

							
2. List of benevolently behaving AV products in test "2001-10":
===============================================================
In general, few scanners could be tested without any problem. When 
considering the large number of postscans, few products had only
minor problems and are regarded as "relatively benevolent".
 
Such "benevolent behaviour" can be reported only for a minority
of DOS scanners:
    ==========================================
    Out of 14 DOS scanners, 7 had NO problems
        and didNOT require any postscan:
    ------------------------------------------
        AVG, AVP, DRW, MR2, NAV, RAV, VSP
    ==========================================
    
In comparison, stability of W-98 scanners improved significantly:
    =============================================================
    Out of 21 W-98 scanners, 8 had NO problems:
    -------------------------------------------
        AVA, AVK, AVP, AVX, DRW, NVC, PAV, RAV
    ============================================

Concerning W-NT related scanners: 
    ==================================================
    Out of 22 W-NT scanners, 8 had NO problems: 
    ------------------------------------------- 
    AVK, AVP, DRW, FPR, FPW, NAV, NVC, PAV, RAV
    ==================================================

Windows-2000 based scanners are the least stable products:
    ==================================================
    Out of 18 W-2000 scanners, 5 had NO problems:
    --------------------------------  ---------
            AVK, AVP, NVC, PAV, RAV       
    ==============================================

Finally, concerning Linux-based scanners:
    ------------------------------------------------
    Out of 9 Linux scanners, only 1 had NO problems:
                      RAV
    ================================================


Concerning overall stability, almost all products required at least
one postscan. As this may be due to the FF/FN anomaly (for which
Microsoft is essentially responsible), the following adresses those
scanners which had NO OTHER problem than requiring postscans:

	***************************************************************
	The following 4 products submitted for ALL 5 platforms behaved
        with least problems (no crash, "only postscans") on ALL
        platforms:                                   AVP, DRW, RAV, SCN
        ***************************************************************
        The following 3 products submitted for 4 platforms (DOS, W98, 
        WNT, W2k) behaved with least problems (no crash, "only 
        postscans") on all 4 platforms for which they were submitted:
                                                          MR2, PAV, VSP
	***************************************************************
        The following 1 product  submitted for 3 platforms (W98, 
        WNT, W2k) behaved with least problems (no crash, "only 
        postscans") on all platforms for which it was submitted:
                                                                    NVC
	***************************************************************
	The following 1 product  submitted for 2 platforms (W98, WNT 
        behaved with least problems (no crash, "only postscans") on all 
        platforms for which it was submitted:
                                                                    AVX
	***************************************************************



3. Problems of AV products observed during test "2001-10":
==========================================================

3.1 List of Postscans:
----------------------
In several cases, AV/AM products didnot access and check all entries 
in testbeds (possibly due to the "FF/FN anomaly" as reported in 1.1) 
or due to crashes or other product misbehaviour (see 3.2). In such cases, 
up to 2 "postscans" were started, whereever possible on the remainder of
the related testbed. 

The following list summarizes those products where at least 1 postscan 
was initialised:

	DOS:	 Macro	     AVA,AVK,PAV,SCN
		 Macro-Pack  AVK,PAV
		 Script	     ANT
		 Script-ITW  ANT

	Linux:	 Macro	     ANT,AVK,AVP,DRW,FSE,SCN,MCV
		 Macro-ITW   MCV
		 Script	     CMD,MCV
		 Script-ITW  MCV

	W-NT:	 Macro	     AVA,IKA,MR2(2x),SCN(2x),VSP
		 Macro-ITW   AVA,MR2
		 Macro-Mal   AVA,FSE
		 Macro-Pack  FSE(2x),INO,SCN
		 Script	     AVA,IKA
		 Script-ITW  AVA,IKA,MR2
		 Script-Mal  IKA

	W-98:	 Macro	     AVG,DSE,MR2,NAV(2x),SCN,VSP
		 Macro-ITW   DSE
		 Macro-Mal   AVG,DSE,FSE,NAV
		 Macro-Pack  AVG,FSE,INO,NAV,QHL
		 Script	     AVG,CMD(2x),DSE,FPR(2x),FPW(2x),FSE,NAV
		 Script-ITW  AVG,DSE,FSE
		 Script-Mal  AVG,DSE,NAV

	W-2k:	 Macro	     AVA,FSE,MCV(2x),R2(2x),NAV(2x),SCN,VSP(2x)
		 Macro-ITW   AVA,MCV
		 Macro-Mal   AVA,NAV,MCV
		 Macro-Pack  AVG,DRW(2x),FSE,INO,MR2,NAV(2x)
		 Script	     AVA,CMD(2x),DRW,FPR(2x),FPW(2x),NAV,MCV
		 Script-ITW  AVA,DRW,MR2,MCV
		 Script-Mal  NAV


3.2 List of specific problems:
------------------------------
The following list reports specific problems observed for products
as indicated ("spoon-feeding" means that scanner was restarted on 
each subsequent directory when a crash was experienced):


    ANT   DOS: one time scanned only install-drive (c:), 
                   although it was commanded to scan t:\irc 
               one time scanned only install-drive (c:), 
                   although it was commanded to scan v:\jvs
          W98: scanner doesn't report files inside testbeds. It simply 
               reports viruses found inside testbeds multiple times.
               Since it is not possible to determine the corresponding 
               file, this was counted as just one detected file per entry, 
               resulting in a detection rate of 284 of 7848 files while the 
               scanner reports a total of 2588 infections.
	       For the same reason, it cannot be assured that all files 
               inside testbeds were really scanned.
          WNT: for packed macro viruses (testbed MACPACK), file is not
               reported, so that an analysis is impossible.
          LIN: ---


    AVA   DOS: ---
          W98: ---
	  WNT: No report was created for 'Macro-Packed' and 'Script-Malware'
               testbeds. Generally, scanner does not report files, that 
               are scanned OK, so it cannot be assured that all files were 
               really scanned in the espective testbeds.
          W2k: ---


    AVG:  DOS: ---
          W98: scanner does not report files, that are scanned OK, so 
               it cannot be assured that all files were really scanned.
          WNT: ---
          W2k: ---


    AVK   DOS: maximum size of 512,000 bytes for report files
          W98: ---
          WNT: ---
          W2k: ---
          LIN: ---


    AVP   DOS: ---
          W98: ---
          WNT: ---
          W2k: ---
          LIN: ---


    AVX:  W98: ---
          WNT: ---


    CMD   DOS: reports file t:\vbs\b\bubblebo\a\vbs_014_.vbs in the summary,
                   but does not report it explicitly as scanned
          W98: one file was not scanned even in 2 postscans: 
                   T:\VBS\B\BUBBLEBO\A\VBS_014_.VBS
          WNT: one file was not scanned even in 2 postscans: 
		   T:\VBS\B\BUBBLEBO\A\VBS_014_.VBS
          W2k: ---
          LIN: ---


    DRA:  WNT: ---



    DRW:  DOS: ---
          W98: ---
          WNT: ---
          W2k: ---
          LIN: ---


    DSE:  W98: scanner does not report files, that are scanned OK, so 
               it cannot be assured that all files were really scanned.


    FPR   DOS: reports the file t:\vbs\b\bubblebo\a\vbs_014_.vbs in the summary,
                   but does not report it explicitly as scanned
          W98: one file was not scanned even in 2 postscans: 
                   T:\VBS\B\BUBBLEBO\A\VBS_014_.VBS
          WNT: ---
          W2k: ---


    FPW:  W98: one file was not scanned even in 2 postscans:
                   T:\VBS\B\BUBBLEBO\A\VBS_004_.VBS
		   T:\VBS\B\BUBBLEBO\A\VBS_005_.VBS
		   T:\VBS\B\BUBBLEBO\A\VBS_014_.VBS
          WNT: ---
          W2k: ---


    FSE:  W98: the following files were reported BOTH as infected 
                   AND as not infected (they were counted as detected):
	           S:\INTENDED\W97M\ANTICOR\A\W97_001_DOC
	       	   S:\TROJAN\XM\RENAMER\B\XM_NONAM.XLS
	       	   T:\VBS\B\BUBBLEBO\A\VBS_013_.VBS
	       	   T:\VBS\C\CHALLENG\A\CHALLN-A.VBS
	       	   T:\VBS\V\VALENTIN\A\VBS_003_.VBS
	       	   V:\VBS\C\CHALLENG\A\CHALLN-A.VBS
	       	   V:\VBS\V\VALENTIN\A\VBS_003_.VBS
          WNT: One file was not reported but counted in the summary:
                   S:\INTENDED\W97M\ANTISOCI\NORMAL.DOT 
               Scanner crashed with message:
		   "error: f-secure management agent not running"
                   while scanning file T:\VBS\B\BUBBLEBO\A\VBS_014_.VBS 
          W2k: ---
          LIN: ---


    IKA:  WNT: ---


    INO:  W98: ---
	  WNT: Some files in the following tstbeds were not scanned:
		   W:\W97M\ETHAN\A\LZH.LZH
		   W:\W97M\STORY\A\LZH.LZH
		   W:\WM\CAP\A\LZH.LZH
		   W:\WM\CONCEPT\A\LZH.LZH NAV
               Scanner does not report files, that are scanned OK, so 
               it cannot be assured that all files were really scanned.
          W2k: ---


    MCV:  1) Product W2kScan for W32, tested for the 1st time
	  2) Product Virscan for Linux, tested for the 1st time
	  W98: Program could not be tested. Following error messages occured:
                   #1 The MAIN.EXE file is linked to missing export 
                       MFC42.DLL:6930.
		   #2 $path$\main.exe  A device attached to the system 
                      is not functioning.
          WNT: Report were only generated when at least one virus was found.
               The following error message appeared (reproducible) when
                   MacroMalware testbed was scanned:
                       "main.exe has generated errors and will be closed 
                        by Windows. You will need to restart the program.
                        An error log is being created. OK"
                   BUT: an error log was NOT found.
                   Postscans of single S:\directories were successful.
          W2k: ---
          LIN: This product had to be compiled on our test platform, 
                   because we only received the source code.
	        Scanner does not report files, that are scanned OK, so 
                   it cannot be assured that all files were really scanned.
                Scanner often reports "cannot open". Very likely, many
                   files have NOT been scanned. 


    MR2:  DOS: ---
          W98: ---
          WNT: ---
          W2k: ---


    NAV:  DOS: ---
          W98: scanner does not report files, that are scanned OK, so 
                   it cannot be assured that all files were really scanned.
          WNT: ---
          W2k: ---


    NVC:  W98: ---
          WNT: ---
          W2k: ---


    PAV:  DOS: ---
          W98: ---
          WNT: ---
          W2k: ---

  
    QHL:  W98: SAME problems as in last test: many crashes for several 
                   testbeds observed: ScriptVir, ScriptVir-ITW: whenever
                   an infected object was found, scanner crashed and could
                   only be finished via TaskManager. No results available 
                   where more than 3 crashes.
		   Remark: product detected macro viruses only when packed.
          WNT: for all script testbeds (script zoo and ITW viruses, script 
                   malware), product scans first sample file endlessly and 
                   does not continue to scan any further file.


    RAV:  DOS: ---
          W98: ---
          WNT: ---
          W2k: ---
          LIN: ---


    SCN:  DOS: ---
          W98: ---
          WNT: ---
          W2k: ---
          LIN: ---


    VSP:  DOS: ---
          W98: ---
          WNT: ---
          W2k: ---
   



	    
