1$FreeBSD: stable/9/tools/regression/lib/libc/nss/README 168754 2007-04-15 11:02:31Z bushman $
2
3A brief how-to
4--------------
5
6Each nsswitch regression test does 2 kinds of actions:
71. It runs a series of queries and tests the correctness of results.
8   There are 2 basic criterias which are used for that:
9      - numbers must be in the correct range
10      - certain pointers should not be NULL
11
122. It makes a snapshot of the results of all queries that were made.
13   The idea of snapshots is to test that nsswitch-related function
14   calls behave equally (i.e. return same results for the same queries)
15   between system upgrades. When the test is executed and the snapshot is
16   already created, the test will compare the result of each query with
17   the appropriate result from the snapshot and will signal if they
18   differ.
19
20In order for nsswitch tests to be as useful as possible you should use
21them in the following way:
22
23Step 1 (before upgrading your system).
24Build the tests with "make" command and execute them with "prove -v"
25command. If there are errors during the execution, then appropriate
26nsswitch functions should be checked. Note, that errors on this state
27can happen only if the particular function return incorrect data.
28
29After the stage 1 a number of "snapshot_[test name]" files will appear
30in your test folder.
31
32Step 2 (after upgrading you system).
33Rebuild the tests with "make clean; make" command and execute them
34again with "prove -v" (check that "snapshot_[test name]" files
35are in the same folder with tests). On this stage regression tests
36will catch not only the correctness errors, but will also determine
37the changes in nsswitch functions behaviour.
38
39In case of the test failure you will get the following message:
40
41To get more details about the error you should do the following:
42Step 1. Run the test alone with debug output enabled.
43Step 2. Mail the snapshot file and the debug test output to the
44freebsd-current@ mailing list.
45
46Example testing session for getpwXXX() family of functions
47----------------------------------------------------------
481. make
49
502. prove -v ./test-getpw.t
51
52   test-getpw....1..8
53   ok 1 - getpwnam()
54   ok 2 - getpwuid()
55   ok 3 - getpwent()
56   ok 4 - getpwent() 2-pass
57   ok 5 - building snapshot, if needed
58   ok 6 - getpwnam() snapshot
59   ok 7 - getpwuid() snapshot
60   ok 8 - getpwent() snapshot
61   ok
62   All tests successful.
63   Files=1, Tests=8,  1 wallclock secs ( 0.00 cusr +  0.20 csys =  0.20 CPU)
64
65
663. Upgrading the system.
67
684. make clean; make
69
705. prove -v ./test-getpw.t (suppose that something has gone wrong)
71
72   test-getpw....1..8
73   ok 1 - getpwnam()
74   ok 2 - getpwuid()
75   ok 3 - getpwent()
76   ok 4 - getpwent() 2-pass
77   ok 5 - building snapshot, if needed
78   not ok 6 - getpwnam() snapshot
79   ok 7 - getpwuid() snapshot
80   not ok 8 - getpwent() snapshot
81   FAILED tests 6, 8
82           Failed 2/8 tests, 75.00% okay
83   Failed 1/1 test scripts, 0.00% okay. 2/8 subtests failed, 75.00% okay.
84
856. We see that test number 6 failed. According to get-getpw.t, this test
86   is executed like this:
87   do_test 6 'getpwnam() snapshot'           '-n -s snapshot_pwd'
88
89   To determine why the test has failed, we need to run it in debug mode -
90   it means adding "-d" to the options list.
91
927. ./test-getpw -dn -s snapshot_pwd
93   ...
94   testing getpwnam() with the following data:
95   toor:*:0:0:0::ne-again Superuser:/root::0:4831
96   testing correctness with the following data:
97   toor:*:0:0:0::Bourne-again Superuser:/root::0:4831
98   correct
99   not ok
100
1018. Here we can see that the data from snapshot (first "toor" line) and
102   the data received from the getpwnam() call (second "toor" line) are
103   different. It is the reason why the test has failed. If you can't
104   (or don't want) to investigate the problem by yourself, mail
105   the test debug output and the snapshot file to the developers list.
106
107Notes on using standalone nsswitch tests
108----------------------------------------
109
110All nsswitch tests have [-d] optional command line argument which enables
111debug output. The debug output can be extremely helpful to determine the
112cause of test failure.
113
114In all nsswitch tests -s <file> command line argument specifies the
115snapshot file. If this file doesn't exist, it would be built during
116test execution. If it already exists then it will be used to check
117the results of particular function calls. This argument is mostly
118optional, but some tests (test-getaddr and test-getusershell) force
119it to be specified.
120
121test-gethostby and test-getaddr require the list of hostnames, that should
122be queried during the test. This list must be specified via -f <file>
123command line argument. Each hostname should occupy exactly one line
124in the file.
125
126Detailed tests description
127--------------------------
128
129./test-getaddr - tests the getaddrinfo() function.
130  Usage: test-getaddr [-d] [-46] [-s <file>] -f <file>
131    -d - enable debug output
132    -4 - force IPv4 usage
133    -6 - force IPv6 usage
134    -s - build/use specified snapshot file
135    -f - use specified hostnames list for testing
136
137./test-getgr
138  Usage: test-getgr -nge2 [-d] [-s <file>]
139    -d - enable debug output
140    -n - test getgrnam(3)
141    -g - test getgrgid(3)
142    -e - test getgrent(3)
143    -2 - test getgrent(3) in 2-pass mode
144    -s - build/use specified snapshot file
145
146./test-gethostby
147  Usage: test-gethostby -na2i [-o] [-d] [-m46] [-s <file>] -f <file>
148    -n - test gethostbyname2(3)
149    -a - test gethostbyaddr(3)
150    -2 - test gethostbyname2(3) results to be equal with getaddrinfo(3)
151         results for the similar query
152    -i - test gethostbyaddr(3) results to be equal with  getnameinfo(3)
153         results for the similar query
154    -o - use getipnodebyname(3)/getipnodebyaddr(3) for testing instead of
155         gethostbyname2(3)/gethostbyaddr(3)
156    -d - enable debug output
157    -m - force IPv4-to-IPv6 mapping
158    -4 - force IPv4 usage
159    -6 - force IPv6 usage
160    -s - build/use specified snapshot file
161    -f - use specified hostnames list for testing
162
163./test-getproto
164  Usage: test-getproto -nve2 [-d] [-s <file>]
165    -d - enable debug output
166    -n - test getprotobyname(3)
167    -v - test getprotobynumber(3)
168    -e - test getprotoent(3)
169    -2 - test getprotoent(3) in 2-pass mode
170    -s - build/use specified snapshot file
171
172./test-getpw
173  Usage: test-getpw -nue2 [-d] [-s <file>]
174    -d - enable debug output
175    -n - test getpwnam(3)
176    -u - test getpwuid(3)
177    -e - test getpwent(3)
178    -2 - test getpwent(3) in 2-pass mode
179    -s - build/use snapshot file
180
181./test-getrpc
182  Usage: test-getrpc -nve2 [-d] [-s <file>]
183    -d - enable debug output
184    -n - test getrpcbyname(3)
185    -v - test getrpcbynumber(3)
186    -e - test getrpcent(3)
187    -2 - test getrpcent(3) in 2-pass mode
188    -s - build/use specified snapshot file
189
190./test-getserv
191  Usage: test-getserv -npe2 [-d] [-s <file>]
192    -d - enable debug output
193    -n - test getservbyname(3)
194    -p - test getservbyport(3)
195    -e - test getservent(3)
196    -2 - test getservent(3) in 2-pass mode
197    -s - build/use specified snapshot file
198
199./test-getusershell
200  Usage: test-getusershell [-d] -s <file>
201    -d - enable debug output
202    -s - build/use specified snapshot file
203
204