From 848d50dc0f0a5c8c28a8a8c4b9ddd1c65176e685 Mon Sep 17 00:00:00 2001 From: Francois Gouget Date: Tue, 17 Sep 2002 00:07:03 +0000 Subject: [PATCH] Start of the conformance testing documentation. Still very incomplete. --- documentation/patches.sgml | 2 +- documentation/running.sgml | 2 +- documentation/testing.sgml | 351 ++++++++++++++++++++++++++++++++++ documentation/wine-devel.sgml | 2 + documentation/wine-doc.sgml | 2 + 5 files changed, 357 insertions(+), 2 deletions(-) create mode 100644 documentation/testing.sgml diff --git a/documentation/patches.sgml b/documentation/patches.sgml index 114d7b3ab0c..2c6fa7a6132 100644 --- a/documentation/patches.sgml +++ b/documentation/patches.sgml @@ -41,7 +41,7 @@ include them as either separate attachments or by appending the diff -u /dev/null /my/new/file output of them to any cvs diff -u output you may have. - Alernatively, use diff -Nu olddir/ newdir/ + Alternatively, use diff -Nu olddir/ newdir/ in case of multiple new files to add. diff --git a/documentation/running.sgml b/documentation/running.sgml index 52aa538e85f..e49af31381c 100644 --- a/documentation/running.sgml +++ b/documentation/running.sgml @@ -253,7 +253,7 @@ winspoolwnetx11 $ wine --dll setupx=n foo.exe - See the DLL chapter for more details. + See the DLL chapter for more details. diff --git a/documentation/testing.sgml b/documentation/testing.sgml new file mode 100644 index 00000000000..052ae7c7cda --- /dev/null +++ b/documentation/testing.sgml @@ -0,0 +1,351 @@ + + Writing Conformance tests + + + Note: This part of the documentation is still very much a work in + progress and is in no way complete. + + + + Introduction + + With more The Windows API follows no standard, it is itself a defacto + standard, and deviations from that standard, even small ones, often + cause applications to crash or misbehave in some way. Furthermore + a conformance test suite is the most accurate (if not necessarily + the most complete) form of API documentation and can be used to + supplement the Windows API documentation. + + + Writing a conformance test suite for more than 10000 APIs is no small + undertaking. Fortunately it can prove very useful to the development + of Wine way before it is complete. + + + + The conformance test suite must run on Windows. This is + necessary to provide a reasonable way to verify its accuracy. + Furthermore the tests must pass successfully on all Windows + platforms (tests not relevant to a given platform should be + skipped). + + + A consequence of this is that the test suite will provide a + great way to detect variations in the API between different + Windows versions. For instance, this can provide insights + into the differences between the, often undocumented, Win9x and + NT Windows families. + + + However, one must remember that the goal of Wine is to run + Windows applications on Linux, not to be a clone of any specific + Windows version. So such variations must only be tested for when + relevant to that goal. + + + + + Writing conformance tests is also an easy way to discover + bugs in Wine. Of course, before fixing the bugs discovered in + this way, one must first make sure that the new tests do pass + successfully on at least one Windows 9x and one Windows NT + version. + + + Bugs discovered this way should also be easier to fix. Unlike + some mysterious application crashes, when a conformance test + fails, the expected behavior and APIs tested for are known thus + greatly simplifying the diagnosis. + + + + + To detect regressions. Simply running the test suite regularly + in Wine turns it into a great tool to detect regressions. + When a test fails, one immediately knows what was the expected + behavior and which APIs are involved. Thus regressions caught + this way should be detected earlier, because it is easy to run + all tests on a regular basis, and easier to fix because of the + reduced diagnosis work. + + + + + Tests written in advance of the Wine development (possibly even + by non Wine developpers) can also simplify the work of the + futur implementer by making it easier for him to check the + correctness of his code. + + + + + FIXME: Ports to new architectures involve New ports involve modifying core parts of Wine: + synchronization, exception handling, thread management, etc. + After such modifications extensive testing is necessary to + make sure the changes did not To check the correctness of new ports. + + + + + + + + What to test for? + + The first thing to test for is the documented behavior of APIs + and such as CreateFile. For instance one can create a file using a + long pathname, check that the behavior is correct when the file + already exists, try to open the file using the corresponding short + pathname, convert the filename to Unicode and try to open it using + CreateFileW, and all other things which are documented and that + applications rely on. + + + While the testing framework is not specifically geared towards this + type of tests, it is also possible to test the behavior of Windows + messages. To do so, create a window, preferably a hidden one so that + it does not steal the focus when running the tests, and send messages + to that window or to controls in that window. Then, in the message + procedure, check that you receive the expected messages and with the + correct parameters. + + + For instance you could create an edit control and use WM_SETTEXT to + set its contents, possibly check length restrictions, and verify the + results using WM_GETTEXT. Similarly one could create a listbox and + check the effect of LB_DELETESTRING on the list's number of items, + selected items list, highlighted item, etc. + + + However, undocumented behavior should not be tested for unless there + is an application that relies on this behavior, and in that case the + test should mention that application, or unless one can strongly + expect applications to rely on this behavior, typically APIs that + return the required buffer size when the buffer pointer is NULL. + + + + + Why have both Perl and C tests? + + + + + + Running the tests on Windows + + The simplest way to run the tests in Wine is to type 'make test' in + the Wine sources top level directory. This will run all the Wine + conformance tests. + + + The tests for a specific Wine library are located in a 'tests' + directory in that library's directory. Each test is contained in a + file, either a '.pl' file (e.g. dlls/kernel/tests/atom.pl) + for a test written in perl, or a '.c' file (e.g. + dlls/kernel/tests/thread.c) for a test written in C. Each + file itself contains many checks concerning one or more related APIs. + + + So to run all the tests related to a given Wine library, go to the + corresponding 'tests' directory and type 'make test'. This will + compile the C tests, run the tests, and create an + 'xxx.ok' file for each test that passes successfully. + And if you only want to run the tests contained in the + thread.c file of the kernel library, you would do: + +$ cd dlls/kernel/tests +$ make thread.ok + + + + Note that if the test has already been run and is up to date (i.e. if + neither the kernel library nor the thread.c file has + changed since the thread.ok file was created), then make + will say so. To force the test to be re-run, delete the + thread.ok file, and run the make command again. + + + You can also run tests manually using a command similar to the + following: + +$ runtest -q -M kernel32.dll -p kernel32_test.exe.so thread.c +$ runtest -p kernel32_test.exe.so thread.c +thread.c: 86 tests executed, 5 marked as todo, 0 failures. + + The '-P wine' options defines the platform that is currently being + tested; the '-q' option causes the testing framework not to report + statistics about the number of successfull and failed tests. Run + runtest -h for more details. + + + + + Inside a C test + + + When writing new checks you can either modify an existing test file or + add a new one. If your tests are related to the tests performed by an + existing file, then add them to that file. Otherwise create a new .c + file in the tests directory and add that file to the + CTESTS variable in Makefile.in. + + + A new test file will look something like the following: + +#include <wine/test.h> +#include <winbase.h> + +/* Maybe auxiliary functions and definitions here */ + +START_TEST(paths) +{ + /* Write your checks there or put them in functions you will call from + * there + */ +} + + + + The test's entry point is the START_TEST section. This is where + execution will start. You can put all your tests in that section but + it may be better to split related checks in functions you will call + from the START_TEST section. The parameter to START_TEST must match + the name of the C file. So in the above example the C file would be + called paths.c. + + + Tests should start by including the wine/test.h header. + This header will provide you access to all the testing framework + functions. You can then include the windows header you need, but make + sure to not include any Unix or Wine specific header: tests must + compile on Windows. + + + + + You can use trace to print informational messages. Note + that these messages will only be printed if 'runtest -v' is being used. + + trace("testing GlobalAddAtomA"); + trace("foo=%d",foo); + + + + + Then just call functions and use ok to make sure that + they behaved as expected: + + ATOM atom = GlobalAddAtomA( "foobar" ); + ok( GlobalFindAtomA( "foobar" ) == atom, "could not find atom foobar" ); + ok( GlobalFindAtomA( "FOOBAR" ) == atom, "could not find atom FOOBAR" ); + + The first parameter of ok is an expression which must + evaluate to true if the test was successful. The next parameter is a + printf-compatible format string which is displayed in case the test + failed, and the following optional parameters depend on the format + string. + + + It is important to display an informative message when a test fails: + a good error message will help the Wine developper identify exactly + what went wrong without having to add too many other printfs. For + instance it may be useful to print the error code if relevant, or the + expected value and effective value. In that respect, for some tests + you may want to define a macro such as the following: + +#define eq(received, expected, label, type) \ + ok((received) == (expected), "%s: got " type " instead of " type, (label),(received),(expected)) + +... + +eq( b, curr_val, "SPI_{GET,SET}BEEP", "%d" ); + + + + Note + + + + + Handling platform issues + + Some checks may be written before they pass successfully in Wine. + Without some mechanism, such checks would potentially generate + hundred of known failures for months each time the tests are being run. + This would make it hard to detect new failures caused by a regression. + or to detect that a patch fixed a long standing issue. + + + Thus the Wine testing framework has the concept of platforms and + groups of checks can be declared as expected to fail on some of them. + In the most common case, one would declare a group of tests as + expected to fail in Wine. To do so, use the following construct: + +todo_wine { + SetLastError( 0xdeadbeef ); + ok( GlobalAddAtomA(0) == 0 && GetLastError() == 0xdeadbeef, "failed to add atom 0" ); +} + + On Windows the above check would be performed normally, but on Wine it + would be expected to fail, and not cause the failure of the whole + test. However. If that check were to succeed in Wine, it would + cause the test to fail, thus making it easy to detect when something + has changed that fixes a bug. Also note that todo checks are accounted + separately from regular checks so that the testing statistics remain + meaningful. Finally, note that todo sections can be nested so that if + a test only fails on the cygwin and reactos platforms, one would + write: + +todo("cygwin") { + todo("reactos") { + ... + } +} + + + But specific platforms should not be nested inside a todo_wine section + since that would be redundant. + + + When writing tests you will also encounter differences between Windows + 9x and Windows NT platforms. Such differences should be treated + differently from the platform issues mentioned above. In particular + you should remember that the goal of Wine is not to be a clone of any + specific Windows version but to run Windows applications on Unix. + + + So, if an API returns a different error code on Windows 9x and + Windows NT, your check should just verify that Wine returns one or + the other: + +ok ( GetLastError() == WIN9X_ERROR || GetLastError() == NT_ERROR, ...); + + + + If an API is only present on some Windows platforms, then use + LoadLibrary and GetProcAddress to check if it is implemented and + invoke it. Remember, tests must run on all Windows platforms. + Similarly, conformance tests should nor try to correlate the Windows + version returned by GetVersion with whether given APIs are + implemented or not. Again, the goal of Wine is to run Windows + applications (which do not do such checks), and not be a clone of a + specific Windows version. + + + FIXME: What about checks that cause the process to crash due to a bug? + + + + + + + + + diff --git a/documentation/wine-devel.sgml b/documentation/wine-devel.sgml index 3ed8aabac54..65529d449e5 100644 --- a/documentation/wine-devel.sgml +++ b/documentation/wine-devel.sgml @@ -7,6 +7,7 @@ + @@ -38,6 +39,7 @@ &compiling; &debugger; &documentation; + &testing; &patches; &i18n; &tools; diff --git a/documentation/wine-doc.sgml b/documentation/wine-doc.sgml index 87ae33221c9..892f84d6ff4 100644 --- a/documentation/wine-doc.sgml +++ b/documentation/wine-doc.sgml @@ -19,6 +19,7 @@ + @@ -91,6 +92,7 @@ &compiling; &debugger; &documentation; + &testing; &patches; &i18n; &tools;