518 lines
22 KiB
Plaintext
518 lines
22 KiB
Plaintext
<chapter id="testing">
|
|
<title>Writing Conformance tests</title>
|
|
|
|
<sect1 id="testing-intro">
|
|
<title>Introduction</title>
|
|
<para>
|
|
With more The Windows API follows no standard, it is itself a defacto
|
|
standard, and deviations from that standard, even small ones, often
|
|
cause applications to crash or misbehave in some way. Furthermore
|
|
a conformance test suite is the most accurate (if not necessarily
|
|
the most complete) form of API documentation and can be used to
|
|
supplement the Windows API documentation.
|
|
</para>
|
|
<para>
|
|
Writing a conformance test suite for more than 10000 APIs is no small
|
|
undertaking. Fortunately it can prove very useful to the development
|
|
of Wine way before it is complete.
|
|
<itemizedlist>
|
|
<listitem>
|
|
<para>
|
|
The conformance test suite must run on Windows. This is
|
|
necessary to provide a reasonable way to verify its accuracy.
|
|
Furthermore the tests must pass successfully on all Windows
|
|
platforms (tests not relevant to a given platform should be
|
|
skipped).
|
|
</para>
|
|
<para>
|
|
A consequence of this is that the test suite will provide a
|
|
great way to detect variations in the API between different
|
|
Windows versions. For instance, this can provide insights
|
|
into the differences between the, often undocumented, Win9x and
|
|
NT Windows families.
|
|
</para>
|
|
<para>
|
|
However, one must remember that the goal of Wine is to run
|
|
Windows applications on Linux, not to be a clone of any specific
|
|
Windows version. So such variations must only be tested for when
|
|
relevant to that goal.
|
|
</para>
|
|
</listitem>
|
|
<listitem>
|
|
<para>
|
|
Writing conformance tests is also an easy way to discover
|
|
bugs in Wine. Of course, before fixing the bugs discovered in
|
|
this way, one must first make sure that the new tests do pass
|
|
successfully on at least one Windows 9x and one Windows NT
|
|
version.
|
|
</para>
|
|
<para>
|
|
Bugs discovered this way should also be easier to fix. Unlike
|
|
some mysterious application crashes, when a conformance test
|
|
fails, the expected behavior and APIs tested for are known thus
|
|
greatly simplifying the diagnosis.
|
|
</para>
|
|
</listitem>
|
|
<listitem>
|
|
<para>
|
|
To detect regressions. Simply running the test suite regularly
|
|
in Wine turns it into a great tool to detect regressions.
|
|
When a test fails, one immediately knows what was the expected
|
|
behavior and which APIs are involved. Thus regressions caught
|
|
this way should be detected earlier, because it is easy to run
|
|
all tests on a regular basis, and easier to fix because of the
|
|
reduced diagnosis work.
|
|
</para>
|
|
</listitem>
|
|
<listitem>
|
|
<para>
|
|
Tests written in advance of the Wine development (possibly even
|
|
by non Wine developers) can also simplify the work of the
|
|
future implementer by making it easier for him to check the
|
|
correctness of his code.
|
|
</para>
|
|
</listitem>
|
|
<listitem>
|
|
<para>
|
|
Conformance tests will also come in handy when testing Wine on
|
|
new (or not as widely used) architectures such as FreeBSD,
|
|
Solaris x86 or even non-x86 systems. Even when the port does
|
|
not involve any significant change in the thread management,
|
|
exception handling or other low-level aspects of Wine, new
|
|
architectures can expose subtle bugs that can be hard to
|
|
diagnose when debugging regular (complex) applications.
|
|
</para>
|
|
</listitem>
|
|
</itemizedlist>
|
|
</para>
|
|
</sect1>
|
|
|
|
|
|
<sect1 id="testing-what">
|
|
<title>What to test for?</title>
|
|
<para>
|
|
The first thing to test for is the documented behavior of APIs
|
|
and such as CreateFile. For instance one can create a file using a
|
|
long pathname, check that the behavior is correct when the file
|
|
already exists, try to open the file using the corresponding short
|
|
pathname, convert the filename to Unicode and try to open it using
|
|
CreateFileW, and all other things which are documented and that
|
|
applications rely on.
|
|
</para>
|
|
<para>
|
|
While the testing framework is not specifically geared towards this
|
|
type of tests, it is also possible to test the behavior of Windows
|
|
messages. To do so, create a window, preferably a hidden one so that
|
|
it does not steal the focus when running the tests, and send messages
|
|
to that window or to controls in that window. Then, in the message
|
|
procedure, check that you receive the expected messages and with the
|
|
correct parameters.
|
|
</para>
|
|
<para>
|
|
For instance you could create an edit control and use WM_SETTEXT to
|
|
set its contents, possibly check length restrictions, and verify the
|
|
results using WM_GETTEXT. Similarly one could create a listbox and
|
|
check the effect of LB_DELETESTRING on the list's number of items,
|
|
selected items list, highlighted item, etc.
|
|
</para>
|
|
<para>
|
|
However, undocumented behavior should not be tested for unless there
|
|
is an application that relies on this behavior, and in that case the
|
|
test should mention that application, or unless one can strongly
|
|
expect applications to rely on this behavior, typically APIs that
|
|
return the required buffer size when the buffer pointer is NULL.
|
|
</para>
|
|
</sect1>
|
|
|
|
|
|
<sect1 id="testing-wine">
|
|
<title>Running the tests in Wine</title>
|
|
<para>
|
|
The simplest way to run the tests in Wine is to type 'make test' in
|
|
the Wine sources top level directory. This will run all the Wine
|
|
conformance tests.
|
|
</para>
|
|
<para>
|
|
The tests for a specific Wine library are located in a 'tests'
|
|
directory in that library's directory. Each test is contained in a
|
|
file (e.g. <filename>dlls/kernel/tests/thread.c</>). Each
|
|
file itself contains many checks concerning one or more related APIs.
|
|
</para>
|
|
<para>
|
|
So to run all the tests related to a given Wine library, go to the
|
|
corresponding 'tests' directory and type 'make test'. This will
|
|
compile the tests, run them, and create an '<replaceable>xxx</>.ok'
|
|
file for each test that passes successfully. And if you only want to
|
|
run the tests contained in the <filename>thread.c</> file of the
|
|
kernel library, you would do:
|
|
<screen>
|
|
<prompt>$ </>cd dlls/kernel/tests
|
|
<prompt>$ </>make thread.ok
|
|
</screen>
|
|
</para>
|
|
<para>
|
|
Note that if the test has already been run and is up to date (i.e. if
|
|
neither the kernel library nor the <filename>thread.c</> file has
|
|
changed since the <filename>thread.ok</> file was created), then make
|
|
will say so. To force the test to be re-run, delete the
|
|
<filename>thread.ok</> file, and run the make command again.
|
|
</para>
|
|
<para>
|
|
You can also run tests manually using a command similar to the
|
|
following:
|
|
<screen>
|
|
<prompt>$ </>../../../tools/runtest -q -M kernel32.dll -p kernel32_test.exe.so thread.c
|
|
<prompt>$ </>../../../tools/runtest -p kernel32_test.exe.so thread.c
|
|
thread.c: 86 tests executed, 5 marked as todo, 0 failures.
|
|
</screen>
|
|
The '-P wine' options defines the platform that is currently being
|
|
tested. Remove the '-q' option if you want the testing framework
|
|
to report statistics about the number of successful and failed tests.
|
|
Run <command>runtest -h</> for more details.
|
|
</para>
|
|
</sect1>
|
|
|
|
|
|
<sect1 id="testing-windows">
|
|
<title>Building and running the tests on Windows</title>
|
|
<sect2>
|
|
<title>Using pre-compiled binaries</title>
|
|
<para>
|
|
Unfortunately there are no pre-compiled binaries yet. However if
|
|
send an email to the Wine development list you can probably get
|
|
someone to send them to you, and maybe motivate some kind soul to
|
|
put in place a mechanism for publishing such binaries on a regular
|
|
basis.
|
|
</para>
|
|
</sect2>
|
|
<sect2>
|
|
<title>With Visual C++</title>
|
|
<itemizedlist>
|
|
<listitem><para>
|
|
get the Wine sources
|
|
</para></listitem>
|
|
<listitem><para>
|
|
Run msvcmaker to generate Visual C++ project files for the tests.
|
|
'msvcmaker' is a perl script so you may be able to run it on
|
|
Windows.
|
|
<screen>
|
|
<prompt>$ </>./tools/winapi/msvcmaker --no-wine
|
|
</screen>
|
|
</para></listitem>
|
|
<listitem><para>
|
|
If the previous steps were done on your Linux development
|
|
machine, make the Wine sources accessible to the Windows machine
|
|
on which you are going to compile them. Typically you would do
|
|
this using Samba but copying them altogether would work too.
|
|
</para></listitem>
|
|
<listitem><para>
|
|
On the Windows machine, open the <filename>winetest.dsw</>
|
|
workspace. This will load each test's project. For each test there
|
|
are two configurations: one compiles the test with the Wine
|
|
headers, and the other uses the Visual C++ headers. Some tests
|
|
will compile fine with the former, but most will require the
|
|
latter.
|
|
</para></listitem>
|
|
<listitem><para>
|
|
Open the <menuchoice><guimenu>Build</> <guimenu>Batch
|
|
build...</></> menu and select the tests and build configurations
|
|
you want to build. Then click on <guibutton>Build</>.
|
|
</para></listitem>
|
|
<listitem><para>
|
|
To run a specific test from Visual C++, go to
|
|
<menuchoice><guimenu>Project</> <guimenu>Settings...</></>. There
|
|
select that test's project and build configuration and go to the
|
|
<guilabel>Debug</> tab. There type the name of the specific test
|
|
to run (e.g. 'thread') in the <guilabel>Program arguments</>
|
|
field. Validate your change by clicking on <guibutton>Ok</> and
|
|
start the test by clicking the red exclamation mark (or hitting
|
|
'F5' or any other usual method).
|
|
</para></listitem>
|
|
<listitem><para>
|
|
You can also run the tests from the command line. You will find
|
|
them in either <filename>Output\Win32_Wine_Headers</> or
|
|
<filename>Output\Win32_MSVC_Headers</> depending on the build
|
|
method. So to run the kernel 'path' tests you would do:
|
|
<screen>
|
|
<prompt>C:\></>cd dlls\kernel\tests\Output\Win32_MSVC_Headers
|
|
<prompt>C:\dlls\kernel\tests\Output\Win32_MSVC_Headers></>kernel32_test path
|
|
</screen>
|
|
</para></listitem>
|
|
</itemizedlist>
|
|
</sect2>
|
|
<sect2>
|
|
<title>With MinGW</title>
|
|
<para>
|
|
This needs to be documented. The best may be to ask on the Wine
|
|
development mailing list and update this documentation with the
|
|
result of your inquiry.
|
|
</para>
|
|
</sect2>
|
|
<sect2>
|
|
<title>Cross compiling with MinGW on Linux</title>
|
|
<para>
|
|
Here is how to generate Windows executables for the tests straight
|
|
from the comfort of Linux.
|
|
</para>
|
|
<itemizedlist>
|
|
<listitem><para>
|
|
First you need to get the MinGW cross-compiler. On Debian all
|
|
you need to do is type <command>apt-get install mingw32</>.
|
|
</para></listitem>
|
|
<listitem><para>
|
|
If you had already run <command>configure</>, then delete
|
|
<filename>config.cache</> and re-run <command>configure</>.
|
|
You can then run <command>make crosstest</>. To sum up:
|
|
<screen>
|
|
<prompt>$ </><userinput>rm config.cache</>
|
|
<prompt>$ </><userinput>./configure</>
|
|
<prompt>$ </><userinput>make crosstest</>
|
|
</screen>
|
|
</para></listitem>
|
|
<listitem><para>
|
|
If you get an error when compiling <filename>winsock.h</> then
|
|
you probably need to apply the following patch:
|
|
<ulink url="http://www.winehq.com/hypermail/wine-patches/2002/12/0157.html">http://www.winehq.com/hypermail/wine-patches/2002/12/0157.html</>
|
|
</para></listitem>
|
|
</itemizedlist>
|
|
</sect2>
|
|
</sect1>
|
|
|
|
|
|
<sect1 id="testing-test">
|
|
<title>Inside a test</title>
|
|
|
|
<para>
|
|
When writing new checks you can either modify an existing test file or
|
|
add a new one. If your tests are related to the tests performed by an
|
|
existing file, then add them to that file. Otherwise create a new .c
|
|
file in the tests directory and add that file to the
|
|
<varname>CTESTS</> variable in <filename>Makefile.in</>.
|
|
</para>
|
|
<para>
|
|
A new test file will look something like the following:
|
|
<screen>
|
|
#include <wine/test.h>
|
|
#include <winbase.h>
|
|
|
|
/* Maybe auxiliary functions and definitions here */
|
|
|
|
START_TEST(paths)
|
|
{
|
|
/* Write your checks there or put them in functions you will call from
|
|
* there
|
|
*/
|
|
}
|
|
</screen>
|
|
</para>
|
|
<para>
|
|
The test's entry point is the START_TEST section. This is where
|
|
execution will start. You can put all your tests in that section but
|
|
it may be better to split related checks in functions you will call
|
|
from the START_TEST section. The parameter to START_TEST must match
|
|
the name of the C file. So in the above example the C file would be
|
|
called <filename>paths.c</>.
|
|
</para>
|
|
<para>
|
|
Tests should start by including the <filename>wine/test.h</> header.
|
|
This header will provide you access to all the testing framework
|
|
functions. You can then include the windows header you need, but make
|
|
sure to not include any Unix or Wine specific header: tests must
|
|
compile on Windows.
|
|
</para>
|
|
<para>
|
|
You can use <function>trace</> to print informational messages. Note
|
|
that these messages will only be printed if 'runtest -v' is being used.
|
|
<screen>
|
|
trace("testing GlobalAddAtomA");
|
|
trace("foo=%d",foo);
|
|
</screen>
|
|
</para>
|
|
<para>
|
|
Then just call functions and use <function>ok</> to make sure that
|
|
they behaved as expected:
|
|
<screen>
|
|
ATOM atom = GlobalAddAtomA( "foobar" );
|
|
ok( GlobalFindAtomA( "foobar" ) == atom, "could not find atom foobar" );
|
|
ok( GlobalFindAtomA( "FOOBAR" ) == atom, "could not find atom FOOBAR" );
|
|
</screen>
|
|
The first parameter of <function>ok</> is an expression which must
|
|
evaluate to true if the test was successful. The next parameter is a
|
|
printf-compatible format string which is displayed in case the test
|
|
failed, and the following optional parameters depend on the format
|
|
string.
|
|
</para>
|
|
</sect1>
|
|
|
|
<sect1 id="testing-error-messages">
|
|
<title>Writing good error messages</title>
|
|
<para>
|
|
The message that is printed when a test fails is
|
|
<emphasis>extremely</> important.
|
|
</para>
|
|
<para>
|
|
Someone will take your test, run it on a Windows platform that
|
|
you don't have access to, and discover that it fails. They will then
|
|
post an email with the output of the test, and in particular your
|
|
error message. Someone, maybe you, will then have to figure out from
|
|
this error message why the test failed.
|
|
</para>
|
|
<para>
|
|
If the error message contains all the relevant information that will
|
|
be easy. If not, then it will require modifying the test, finding
|
|
someone to compile it on Windows, sending the modified version to the
|
|
original tester and waiting for his reply. In other words, it will
|
|
be long and painful.
|
|
</para>
|
|
<para>
|
|
So how do you write a good error message? Let's start with an example
|
|
of a bad error message:
|
|
<screen>
|
|
ok(GetThreadPriorityBoost(curthread,&disabled)!=0,
|
|
"GetThreadPriorityBoost Failed");
|
|
</screen>
|
|
This will yield:
|
|
<screen>
|
|
thread.c:123: Test failed: GetThreadPriorityBoost Failed
|
|
</screen>
|
|
</para>
|
|
<para>
|
|
Did you notice how the error message provides no information about
|
|
why the test failed? We already know from the line number exactly
|
|
which test failed. In fact the error message gives strictly no
|
|
information that cannot already be obtained by reading the code. In
|
|
other words it provides no more information than an empty string!
|
|
</para>
|
|
<para>
|
|
Let's look at how to rewrite it:
|
|
<screen>
|
|
BOOL rc;
|
|
...
|
|
rc=GetThreadPriorityBoost(curthread,&disabled);
|
|
ok(rc!=0 && disabled==0,"rc=%d error=%ld disabled=%d",
|
|
rc,GetLastError(),disabled);
|
|
</screen>
|
|
This will yield:
|
|
<screen>
|
|
thread.c:123: Test failed: rc=0 error=120 disabled=0
|
|
</screen>
|
|
</para>
|
|
<para>
|
|
When receiving such a message, one would check the source, see that
|
|
it's a call to GetThreadPriorityBoost, that the test failed not
|
|
because the API returned the wrong value, but because it returned an
|
|
error code. Furthermore we see that GetLastError() returned 120 which
|
|
winerror.h defines as ERROR_CALL_NOT_IMPLEMENTED. So the source of
|
|
the problem is obvious: this Windows platform (here Windows 98) does
|
|
not support this API and thus the test must be modified to detect
|
|
such a condition and skip the test.
|
|
</para>
|
|
<para>
|
|
So a good error message should provide all the information which
|
|
cannot be obtained by reading the source, typically the function
|
|
return value, error codes, and any function output parameter. Even if
|
|
more information is needed to fully understand a problem,
|
|
systematically providing the above is easy and will help cut down the
|
|
number of iterations required to get to a resolution.
|
|
</para>
|
|
<para>
|
|
It may also be a good idea to dump items that may be hard to retrieve
|
|
from the source, like the expected value in a test if it is the
|
|
result of an earlier computation, or comes from a large array of test
|
|
values (e.g. index 112 of _pTestStrA in vartest.c). In that respect,
|
|
for some tests you may want to define a macro such as the following:
|
|
<screen>
|
|
#define eq(received, expected, label, type) \
|
|
ok((received) == (expected), "%s: got " type " instead of " type, (label),(received),(expected))
|
|
|
|
...
|
|
|
|
eq( b, curr_val, "SPI_{GET,SET}BEEP", "%d" );
|
|
</screen>
|
|
</para>
|
|
</sect1>
|
|
|
|
|
|
<sect1 id="testing-platforms">
|
|
<title>Handling platform issues</title>
|
|
<para>
|
|
Some checks may be written before they pass successfully in Wine.
|
|
Without some mechanism, such checks would potentially generate
|
|
hundred of known failures for months each time the tests are being run.
|
|
This would make it hard to detect new failures caused by a regression.
|
|
or to detect that a patch fixed a long standing issue.
|
|
</para>
|
|
<para>
|
|
Thus the Wine testing framework has the concept of platforms and
|
|
groups of checks can be declared as expected to fail on some of them.
|
|
In the most common case, one would declare a group of tests as
|
|
expected to fail in Wine. To do so, use the following construct:
|
|
<screen>
|
|
todo_wine {
|
|
SetLastError( 0xdeadbeef );
|
|
ok( GlobalAddAtomA(0) == 0 && GetLastError() == 0xdeadbeef, "failed to add atom 0" );
|
|
}
|
|
</screen>
|
|
On Windows the above check would be performed normally, but on Wine it
|
|
would be expected to fail, and not cause the failure of the whole
|
|
test. However. If that check were to succeed in Wine, it would
|
|
cause the test to fail, thus making it easy to detect when something
|
|
has changed that fixes a bug. Also note that todo checks are accounted
|
|
separately from regular checks so that the testing statistics remain
|
|
meaningful. Finally, note that todo sections can be nested so that if
|
|
a test only fails on the cygwin and reactos platforms, one would
|
|
write:
|
|
<screen>
|
|
todo("cygwin") {
|
|
todo("reactos") {
|
|
...
|
|
}
|
|
}
|
|
</screen>
|
|
<!-- FIXME: Would we really have platforms such as reactos, cygwin, freebsd & co? -->
|
|
But specific platforms should not be nested inside a todo_wine section
|
|
since that would be redundant.
|
|
</para>
|
|
<para>
|
|
When writing tests you will also encounter differences between Windows
|
|
9x and Windows NT platforms. Such differences should be treated
|
|
differently from the platform issues mentioned above. In particular
|
|
you should remember that the goal of Wine is not to be a clone of any
|
|
specific Windows version but to run Windows applications on Unix.
|
|
</para>
|
|
<para>
|
|
So, if an API returns a different error code on Windows 9x and
|
|
Windows NT, your check should just verify that Wine returns one or
|
|
the other:
|
|
<screen>
|
|
ok ( GetLastError() == WIN9X_ERROR || GetLastError() == NT_ERROR, ...);
|
|
</screen>
|
|
</para>
|
|
<para>
|
|
If an API is only present on some Windows platforms, then use
|
|
LoadLibrary and GetProcAddress to check if it is implemented and
|
|
invoke it. Remember, tests must run on all Windows platforms.
|
|
Similarly, conformance tests should nor try to correlate the Windows
|
|
version returned by GetVersion with whether given APIs are
|
|
implemented or not. Again, the goal of Wine is to run Windows
|
|
applications (which do not do such checks), and not be a clone of a
|
|
specific Windows version.
|
|
</para>
|
|
<!--para>
|
|
FIXME: What about checks that cause the process to crash due to a bug?
|
|
</para-->
|
|
</sect1>
|
|
|
|
|
|
<!-- FIXME: Strategies for testing threads, testing network stuff,
|
|
file handling, eq macro... -->
|
|
|
|
</chapter>
|
|
|
|
<!-- Keep this comment at the end of the file
|
|
Local variables:
|
|
mode: sgml
|
|
sgml-parent-document:("wine-devel.sgml" "set" "book" "part" "chapter" "")
|
|
End:
|
|
-->
|