===================================
File 8PROBLMS.TXT:
-----------------------------------
List of problems experienced during 
     aVTC test "2002-12":
===================================
Formatted with non-proportional font (Courier)


Content of this file:
=====================
    1. Introduction: General Problems 
       1.1 Problems likely related to FindFirst/FindNext anomaly
           (essentially unchanged since last test)
    2. List of benevolently behaving AV products in test "2002-12"
    3. Problems of AV products observed during test "2002-12"
       3.1 List of Postscans
       3.2 List of specific problems


1. Introduction: General Problems:
==================================
For automatic tests on large viral databases, and for automatic 
processing of large scanner log files, a set of test conditions 
is prerequisite for scanners to participate in a VTC test (see: 
4TESTCON.TXT).

In many cases, serious problems were observed during some tests.
DOS scanners were either not suitably running under SIMBOOT and 
crashed, or problems appeared with the (rather large) file virus 
database. In some cases, scanners crashed upon detecting some 
specific virus; in few cases, "manual" operation instead of 
automatic (batch) operation helped solving some of these problems. 
Such curative action was also applied when possible in cases where 
log files were inadequate (e.g.needing manual operation for export).

With growing velocity of processors, DOS scanners (running without 
any problem on INTEL 386 and 486) growingly crash on Pentium II/III
systems faster than 250 MHz. Another general problem with DOS scanners 
is related to counters for files and viruses which often seem to be 
designed as integers, so they start after 65,536 with 0.

During preparation and test, we again experienced a serious problem 
reported also in previous VTC Tests, according to which management of
large sets of directories in FAT and NTFS may not reliably work. Both
when attempting to move large parts of our file virus database, as 
when some scanner proceeded scanning subsequent viral directories, 
we found that several directories were not moved or touched. This
effect seems to happen stochastically, such that subsequent attempts
gave different results. Concerning omitted (=unscanned) directories,
we overcame this "dysfunctional" behaviour of FAT and NTFS by repeat-
ing scanning so long until the number of scanned files agreed with 
the (known) number of directories in testbeds. Overcoming this  
problem was extremely time-consuming, and this was a reason for 
delaying publication of results. 

In cases where scanners crashed during detection test upon the rather
large file virus database, tests were performed in several runs on
partitions (essentially on directories with same first letters of
names). In most cases (apart those reported below), these tests
were completed, and resulting files were joined and evaluated.

Finally, with growing testbeds, test protocols produced by scanners
grow equally. When processing such protocols, we need meanwhile up
to 6 GByte of disk space, and our evaluation scripts (in AWK) 
become more complex. Under these conditions, we also suffered from
an evident bug in the AWK processor which inhibited proper evaluation
and required additional quality assurance (including time and efforts). 


1.1 Problems very likely related to FindFirst/FindNext anomaly:
---------------------------------------------------------------
In several cases, scanners finished a first scan although they
had not touched all directories with infected objects. In such
a case, a postscan was started adressing only those untouched
objects; a second postscan was started when again objects were
observed untouched, but after the 2nd postscan, no more scan
was started. This behaviour may originate from a reported anomaly
in the behaviour of FindFirst/FindNext (those routines are used
to handle objects in directory trees) which has not been cured
so far by Microsoft.

In the "problems list", these postscans are ked as "minor problems":
different from crashes, postscans "only" required time for running
the tests and evaluating test protocols (which significantly delayed
results).

							
2. List of benevolently behaving AV products in test "2002-12":
===============================================================
In general, few scanners could be tested without any problem. When 
considering the large number of postscans, few products had only
minor problems and are regarded as "relatively benevolent".
 
Such "benevolent behaviour" can be reported only for a minority
of DOS scanners:

    A) DOS products:
    ============================================
      ALL 13 DOS scanners had problems (either
          crashing or producing error logs)          
     ============================================
    (Performance in last test: 7 scanners of 14
     without problems)
    

    B) W98 products:
    ============================================
    Out of 18 W-98 scanners, 7 had NO problems:
    -------------------------------------------
        AVA, AVK, AVP, BDF, DRW, FPW, NVC
    ============================================
    (Performance in last test: 8 scanners of 21
     without problems))


    C) W-2000 products:
    ==============================================
    Out of 19 W-2000 scanners, 10 had NO problems:
    ----------------------------------------------
            AVK, AVP, NVC, PAV, RAV       
    ==============================================
    (Performance in last test: 5 scanners of 18
     without problems)


    D) LINUX products:
    --------------------------------------------
    Out of 8 Linux scanners, 5 had NO problems:
             CMD, DRW, FSE, OAV and RAV
    ============================================
    (Performance in last test: 1 scanner of 9
     without problems)

Concerning overall stability, almost all products required at least
one postscan. As this may be due to the FF/FN anomaly (for which
Microsoft is essentially responsible), the following adresses those
scanners which had NO OTHER problem than requiring postscans:

	***************************************************************
	The following product submitted for ALL 4 platforms behaved
        with least problems (no crash, "only postscans") on ALL
        platforms:                                                  DRW
        ***************************************************************
        NO product  submitted for 3 platforms (W98, WNT, W2k) behaved 
        with least problems (no crash, "only postscans") on all 
        platforms for which it was submitted:
                                                                    (0)
	***************************************************************
	The following 4 product2  submitted for 2 platforms (W98, WNT 
        behaved with least problems (no crash, "only postscans") on all 
        platforms for which it was submitted:
                                                     AVK, AVP, BDF, FPW
	***************************************************************



3. Problems of AV products observed during test "2002-12":
==========================================================

3.1 List of Postscans:
----------------------
In several cases, AV/AM products didnot access and check all entries 
in testbeds (possibly due to the "FF/FN anomaly" as reported in 1.1) 
or due to crashes or other product misbehaviour (see 3.2). In such cases, 
up to 2 "postscans" were started, whereever possible on the remainder of
the related testbed. 

The following list summarizes those products where at least 1 postscan 
was initialised (2x implies that 2 postscans were needed):

  DOS:  BOOT-ITW: ---
        FILE-PAC: AVG(2x),AVP,CMD(2x),FPR,INO,NAV(2x),NVC,RAV
        FILE-ITW: INO,NVC
        FILE-MAL: AVA,CMD(2x),FPR(2x),NAV(2x),NVC,RAV(2x),SCN,VSP(2x)
            FILE: AVP(2x),CMD(2x),FPR(2x),INO(2x),NAV(2x),SCN,VSP(2x)
        MACR-PAC: AVG,AVP,INO,NAV,RAV
        MACR-ITW: ---
        MACR-MAL: AVA,INO,NAV(2x)
            MACR: AVA(2x),INO,NAV,SCN,VSP(2x)
        SCRI-ITW: INO
        SCRI-MAL: INO,NAV
            SCRI: CMD(2x),FPR(2x),INO,NAV

  LNX:  FILE-PAC: CMD,FPR,RAV(2x),SCN(2x)
        FILE-ITW: CMD,FPR
        FILE-MAL: AVK(2x),CMD,DRW,FPR(2x),FSE,OAV,RAV,SCN
            FILE: AVK(2x),CMD(2x),DRW,FPR(2x),FSE,SCN
        MACR-PAC: AVK(2x),RAV
        MACR-ITW: DRW,FSE,RAV
        MACR-MAL: ---
            MACR: AVK(2x),CMD,DRW,FPR(2x),FSE,OAV,RAV,SCN
        SCRI-ITW: ---
        SCRI-MAL: ---
            SCRI: CMD,FPR

  W2K:  BOOT-ITW: 
        FILE-PAC: AVG(2x),AVK,AVP,CMD,FPR,FPW,FSE,IKA(2x),INO,PRO(2x),RAV(2x)
        FILE-ITW: AVP,CMD,INO,NVC,PRO,RAV
        FILE-MAL: AVP,CMD(2x),FPW,INO(2x),MR2(2x),NVC(2x),PRO(2x),RAV,VSP
            FILE: AVP,CMD,FPR(2x),FPW,INO(2x),MR2(2x),NVC,PRO,RAV(2x),VSP(2x)
        MACR-PAC: AVG(2x),AVK,AVP(2x),BDF,FPW,FSE(2x),IKA,INO,NAV,PRO,RAV(2x)
        MACR-ITW: FPW,PRO,RAV
        MACR-MAL: FPW,INO,MR2,NAV,PRO,RAV
            MACR: FPW,INO(2x),MR2(2x),NAV(2x),PRO,RAV,VSP
        SCRI-ITW: INO(2x),MR2,PRO
        SCRI-MAL: FPW,INO,MR2,NAV,PRO,RAV
            SCRI: CMD,FPR(2x),FPW(2x),INO,MR2(2x),NAV,PRO,RAV

  W98:  FILE-PAC: AVG(2x),AVK(2x),AVP,CMD,FPR,FPW(2x),FSE(2x),INO,NAV,NVC(2x),PRO,RAV
        FILE-ITW: AVP,BDF,FSE(2x),INO,NVC(2x),PRO
        FILE-MAL: AVP,FPW,FSE(2x),INO(2x),NVC(2x),PRO,RAV,VSP
            FILE: AVA,AVP(2x),FPR,FPW,FSE(2x),INO(2x),NAV(2x),NVC(2x),PRO(2x),VSP(2x)
        MACR-PAC: AVG(2x),AVK,AVP(2x),BDF,FPW,FSE(2x),INO,NAV,PRO,RAV(2x)
        MACR-ITW: BDF,FPW,PRO,RAV
        MACR-MAL: BDF,FPW,FSE,INO,NAV,PRO,RAV
            MACR: AVP,FPW,FSE(2x),INO(2x),MR2(2x),NAV(2x),PRO,RAV,VSP
        SCRI-ITW: BDF,INO,PRO
        SCRI-MAL: BDF,FPW,FSE(2x),INO,NAV,PRO,RAV
            SCRI: BDF,CMD,FPR,FPW,FSE(2x),INO,MR2,NAV,PRO,RAV,VSP(2x)


3.2 List of specific problems:
------------------------------
The following list reports specific problems observed for products
as indicated ("spoon-feeding" means that scanner was restarted on 
each subsequent directory when a crash was experienced):

General problems:
-----------------  
      The WinRAR archives for the "Macro packed" testbed require at least 
      RAR 2.00 to extract files, while the WinRAR archive in the  
      "File packed" testbed require at least RAR 2.90. Therefore most 
      products *failed* to detect the WinRAR archives in "File packed" 
      testbed although the WinRAR archives in the "Macro packed" testbed 
      were (at least partially) detected.


    AVA:  DOS: Crashed three times during boot-ITW test and was 
               excluded from this part of the test.
	  W2K: Didnot scan archives.
          W98: ---


    AVG:  DOS: Crashed two times while scanning file-zoo on 
                 t:\sod\mronon\k\sserpyek\2321\a\exa_005_.exe.
               Crashed three times while scanning file-itw 
                 and was excluded from this part of the test.
               Crashed three times while scanning file-malware 
                 and was excluded from this part of the test.
               Crashed three times while scanning file-pack 
                 and was excluded from this part of the test.
               Crashed three times while scanning boot-itw 
                 and was excluded from this part of the test.
	  W2K: ---
	  W98: While scanning Drive V: (containsing packed file-viruses),
               an error occured at the files
               v:\file\23w\mronon\g\gug\2621\a\wrar.rar       (11)
               v:\file\sod\mronon\o\flah_eno\4453\a\wrar.rar  (20)
               v:\file\23w\mronon\k\zirk\0504\lenrek\wrar.rar (28)
               Error: "CauseWay DOS Extender: Exception: 0E, Error code: 0004"
               The complete error-message for each file is documented in 
               2002_04_w98_v<num>.err.txt (see number at file)


    AVK:  W98: ---
          W2k: ---
          LNX: In the command line, the engine does not accept more 
               than 7 objects to be scanned.


    AVP:  DOS: ---
	  W98: ---
          W2k: ---


    BDF:  W98: ---
          W2k: ---


    CMD:  DOS: ---
          W98: ---
          W2k: ---


    DRW:  DOS: Crashed three times during file-itw scans 
                 and was excluded from this part of the test.
               Crashed three times during file-malware scans 
                 and was excluded from that part of the test.
               Crashed three times during file-zoo scans (once 
                 on t:\sod\mronon\h\cllh\0967\exa_000_.exe 
                 and two times on other files) and was excluded 
                 from this part of the test.
          W98: ---
          W2k: ---
          LNX: ---


    FPR:  DOS: ---
          W2k: ---  
          W98: While scanning drive V: (containing packed file-viruses),
               an error occurs at the files:
      	       V:\MALW\MROW\MRONON\P\KRAPYTTE.RP\33415\WRAR.RAR	  (07)
      	       V:\MALW\MROW\MRONON\W\23W\EKOHC\NEG\MROW\WRAR.RAR  (08)
      	       V:\MALW\MROW\MRONON\W\23W\TINISM\MROW\A\WRAR.RAR	  (09)
               Error: "CauseWay error 09 : Unrecoverable exception. 
                      Program terminated." The complete error-message 
               for each file is documented in 200204_98_v<number>.err.txt 
               (see number at file).
          LNX: When scanning long lists of arguments, as required for 
               postscans, FProt crashes with "segment violation".


    FPW:  W98: ---
          W2k: ---


    FSE:  W98: Some samples are never reported or scanned. It seems 
               that these cases don't occur randomly but always with 
               the last file in a directory. 
               In addition, some files are reported twice: the first 
               time the sample is reported as usual, the second time 
               only the filename is printed. 
               There seems to be an coincidence between both errors, 
               because often the last file in a directory is not 
               reported at all AND the file before is reported twice.
               The last file in _every_ report is always reported twice.
               Macro-ITW-Packed: the first entry in every archive is 
               not reported/scanned at all (altough there are no 
               problems with the unpacked files!).
          W2k: ---
          LNX: ---


    IKA:  W2k: ---


    INO:  General: This product only reports infected files, so we 
               cannot assure that all files were really scanned.
               3 samples in the "packed File" testbed were detected 
               by the heuristic engine on all systems in ARJ and ZIP 
               archives but were NOT be counted in the result, because 
               the engine failed to report the complete filenames.
          DOS: Crashed three times during file-malware test 
               and was excluded from this part of the test
          W98: This product didnot scan the file:
                W:\NAJORT\MRONON\B\SP-ROODK.CAB\EXA_006_.EXE
               in the malware testbed. Reports are cutted before 
               this file without errormessage.
          W2K: This product didnot scan the file:
                W:\NAJORT\MRONON\B\SP-ROODK.CAB\EXA_006_.EXE
               in the malware testbed. Reports are cutted before 
               this file without errormessage.


   MR2:   DOS: Required different keystrokes during boot-testing 
               and produced lot of I/O errors and was canceled 
               for this part of the test.
               Crashed three times during file-zoo scans 
               and was excluded from this part of the test.
      W98/W2K: under all windows systems, this product probably 
               doesn't free memory if started in same DOS-window again. 
               When scanning large testbeds, many files are not scanned 
               but an error is given:
                     "I/O-Fehler, Datei nicht ueberprueft!"
               Therefore, even after 2 postscans not all files were 
               scanned (e.g. W2k - Macro zoo: 17851 files were not scanned).


   NAV:   General: This product only reports infected files, so we cannot
               assure that all files were really scanned.
          DOS: Product crashed on file-mal on the file:
                  W:\NAJORT\MRONON\I\LLEHS-I\EXA_000_.EXE
          W98: Product only scanned the first 826 files in the whole 
               "File Zoo" testbed (but did not crash!). This behaviour could 
               be reproduced on different computers and with different scan 
               methods (scan whole drive and scan single subdirectories).
               Only scanning single subdirectories from last to first 
               (alphabetically) results in more files found, but then 
               product crashed very soon. 
               It was not possible to scan most of the testbeds contents 
               with this product (only the cmdline version 'navdx.exe' 
               was tested).
	  W2K: Some samples in the "Macro zoo" and the "File zoo" testbeds 
               were reported as infected but could NOT be counted no path 
               path and file name were logged but just random characters 
               were written to the logfile.
               In addition, the action is reported as "unknown" instead 
               of "delete", "leave alone", etc. 
               For example:
               "9/10/2002 11:38:14 AM,H,W97M.ANTISOCIAL.F,LEFT ALONE,EXCALIBUR,
                ADMINISTRATOR,H,INFECTED,H,UNKNOWN ACTION,LEAVE ALONE 
                (LOG ONLY),MANUAL SCAN"
               "12/12/2002 6:27:24 PM,3),MTE.INSUF (2,LEFT ALONE,METEOR,
                ADMINISTRATOR,3),INFECTED,3),UNKNOWN ACTION,LEAVE ALONE 
                (LOG ONLY), MANUAL SCAN"
               For the "File packed" testbed, many samples were reported as 
               infected but could NOT be counted because the filename was not 
               reported properly. 
               A total of 99 samples (only in ARJ archives) is not counted 
               for the result. For example:
               "9/24/2002 2:48:08 PM,KSI...,W32.BLEBLA.WORM,FILE; 
                COMPRESSED FILE, LEFT ALONE,EXCALIBUR,ADMINISTRATOR,
                V:\MALW\KSIR_CES\MRONON\W\23W\MM_ALBEL.B\ARJ.ARJ>>MALW\,INFECTED,
                V:\MALW\KSIR_CES\MRONON\W\23W\MM_ALBEL.B\ARJ.ARJ>>MALW\,
                LEAVE ALONE (LOG ONLY),LEAVE ALONE (LOG ONLY),MANUAL SCAN"


    NVC:  General: There are no parameters to create an ASCII-reportfile 
               with NVCC (command-line-scanner for W98, W2K and WXP). 
               Using divert screen-output ">".
	  DOS: Crashed three times 
               on t:\sod\mronon\l\unemel\309\exa_001_.exe.
          W98: ---
          W2k: Altough some files produced seemingly critical error messages, 
               they were nevertheless reported as infected:
                   "Unpacking: COA_001_.COM
                    Cannot open: T:\SOD\MRONON\O\CO\75\COA_001_.COM.
                    Invalid handle
                    T:\SOD\MRONON\O\CO\75\COA_001_.COM
                      Possible virus in 'T:\SOD\MRONON\O\CO\75\COA_001_.COM' 
                      -> 'Austr/Trivial_based'"
                   "Unpacking: COA_000_.COM
                    Cannot open: T:\SOD\MRONON\O\CO\75\COA_000_.COM.
                      NDECOMP: No files unpacked
                      T:\SOD\MRONON\O\CO\75\COA_000_.COM
                      Possible virus in 'T:\SOD\MRONON\O\CO\75\COA_000_.COM' 
                      -> 'Austr/Trivial_based'"


    OAV:  LNX: ---


    PRO:  W98: ---
          W2k: ---


    RAV:  DOS: Crashed three times during boot-itw test 
               and was excluded from this part of the test
               Product only runs on one specific computer (which is 
               quite strange, since all dos-computers in VTC have 
               the same hardware and software)
          W98: While scanning archive j:\WM\CONCEPT\A\WRAR.RAR, Windows reported
               that RAVAV crashed. The following message box was displayed:
                  "This programm has performed an illegal operation 
                   and will be shut down.
                   If the problems persists, contact the program vendor.
               The "Details"-function of the error handler displays:
                   RAVAV caused an invalid page fault in
                   module <unknown> at 0000:00c64253.
                   Registers:
                   EAX=00ca5eb8 CS=015f EIP=00c64253 EFLGS=00010246
                   EBX=00000000 SS=0167 ESP=0064f52c EBP=00000001
                   ECX=415c5450 DS=0167 ESI=415c5450 FS=2fc7
                   EDX=00c665c0 ES=0167 EDI=415c5450 GS=0000
                   Bytes at CS:EIP:
                   8b 46 14 85 c0 7e 28 33 ff 8b 06 8b 04 07 85 c0 
                   Stack dump: 415c5450 00ca5eb8 010e11a0 00c662d4 
                               00000000 00ca5eb8 00c65d2c 010e115c 
                               00c11125 00ca5eb8 010e11a0 00000410 
                               00ca5a08 00000084 00000000 00ca5eb8 
               Product crashed twice upon scanning 
                 W:\mrow\zerolp.xe\mrow\kap\m_c\exa_000_.exe
                 with the following error message: 
                   RAVAV caused an invalid page fault in
                   module <unknown> at 0000:00bc2a2c.
                   Registers:
                   EAX=0002b800 CS=015f EIP=00bc2a2c EFLGS=00010206
                   EBX=00c9e9f8 SS=0167 ESP=0064f248 EBP=00180014
                   ECX=00180014 DS=0167 ESI=0002b800 FS=2117
                   EDX=00000200 ES=0167 EDI=00c079fc GS=0000
                   Bytes at CS:EIP:
                   8b 45 f8 8b 7d fc 3d be ba fe ca 74 07 5f 5e 5d 
                   Stack dump: 00c079fc 00000200 00c9ea28 00c9e9f8 
                               00c10007 00180014 0002b800 000001b4 
                               00c077fc 00e20f59 00c9e9f8 00c077fc 
                               00000200 00f9cb38 00c9ebb8 00c9ea28 
          W2k: Crashed on File zoo testbed while scanning the file: 
                  T:\SOD\MRONON\C\LETPYRC\062\COA_001_.COM
          LNX: ---
      

    SCN:  DOS: ---
          W98: ---
          W2k: Report files generated by SCN for scanning filepack contain
               no winrar archives (and no error messages) under Windows 2K. 
               Error-message on console:
                  <Path to WINRAR-archive>\<internal file> ... 
                        file could not be opened.
               Problem may apply to DOS and W98 as well, because no error 
               messages are recorded in logfile.
          LNX: While scanning "file packed", all WinRar-Archive were logged 
               with the error message: "file could not be opened."


    VSP:  DOS: Crashed three times during file-test 
               and was excluded from this part of the test.
               Product could not be tested in boot test 
               because of multiple necessary keystrokes while running.
               Crashed three times during file-itw test.
               While scanning the file zoo testbed, the error message 
               "Fehler: Handbuch/Bedienungsanleitung fehlt" was displayed.
          W98: ---
          W2k: ---




