This year the PHP Community promoted the PHPTestFest in various places around the globe and it was a success. I participated in the event promoted by the PHPSP UG (PHP Sao Paulo User Group) which ended up as being the team with the most tests committed. After participating in the event I kept going and have since obtained an SVN account and karma to be an official test committer to PHP. If you did not get the chance to learn how to write tests during your local TestFest, I will now go over the steps taken so you can be ready for next year. PHPT is an extremely simple test framework designed to test PHP itself. Its atomic nature and independent test execution makes it perfect for the kind of tests needed, so tests can be really focused on testing specific functions and/or bugs. What do I need to know? The beauty of PHPT is that you need to know very little other than how to write PHP code. A little knowledge of the inner workings of PHP will of course help you in finding areas of code that need testing, and how best to test them, but just knowing PHP is enough to start. I have divided this post into 5 parts:
- Preparing the Environment
- Choosing what to test
- Writing a test
- Executing a test
- Submitting a test to PHP
1. Preparing the Environment
Your test environment needs the correct version of PHP and the ability to run tests, I’m going to focus on *nix/MacOS systems here as I have not had experience with tests on Windows. The first step is to compile PHP from source. During the testfest we focused on PHP 5.3, but you should head over to the PHP QA Page (qa.php.net) to see what version is being targeted, or else take you pick from the PHP SVN. In any case – you need to download the source code that is appropriate for your system. The basic compilation process means you need to execute ./configure with your correct settings, and then run make. After running make you are already ready to run tests using the make test command. The latter will run all the current tests (9000+) and prompt you to send a report to the QA Team. But before you begin writing your own tests I recommend that you also run make install so you can have a proper PHP executable for the version you are testing. Later on we will see how to use make test to target only the tests you wish to execute.
2. Choosing what to test
Before you begin writing a test you must decide on what you are writing a test for. You will usually be either writing a test for a specific bug or to cover areas of PHP without “code coverage”. To figure out the bugs that have/need tests you should get on the QA mailing list or check out the correct path (for example for GD go to ext/gd/tests) for a filename with the bug-id. To write tests to cover untouched areas of the source code your best bet is to head over to http://gcov.php.net where you should pick you branch/version and look for the areas with less code coverage. In this year’s testfest the PHPSP group focused on the GD extension, so we where basically looking at the coverage for the gd.c file, which i will use for examples in this article. To decide what to test took a little bit of bit-scrubbing, so this is why I remarked earlier that knowing the inner working of PHP helps. However – after a few examples you will easily pick up what needs to be done.
Let’s start by analysing these lines of code from gd.c:
What we are looking for here are the red lines, which means that these lines were not executed. The main objective is to get all lines to either blue (executed) or white (non-executable code) as you can see above. The first thing you should notice here is that this is the code for the imageclorallocatealpha function as displayed online 1848 by the PHP_FUNCTION construct. Now you should look at the red lines where we can easily see that line 1856 is never executed, and judging by the IF construct before it we can state that this line is never executed because the zend_parse_parameters function (nevermind what it does for now) never returns the FAILURE state. As it happens, the zend_parse_parameters is responsible for parsing and validating the parameters passed to the function, so we can guess that this means that the current tests never pass an invalid parameter to the function, thus never testing if it actually returns a proper error message. This seems like good place to begin learning about tests.
3. Writing a test
Writing the actual test is usually really simple, as it should be since tests need to be atomic. You should limit the scope of your test to testing only the function at hand and the particular situation you want to test – this will also help you name the test properly, as we will see next.
3.1 Naming test files
Naming test files is simple if you defined you scope correctly:
- Bug test: bug<bugid>.phpt
- Basic function usage: <função>_basic.phpt
- Function error behaviours: <função>_error.phpt
- Variations of function usage: <função>_variation.phpt
- Extensions: <extensão>_<#>.phpt
3.2 Structure of a .phpt file
The simple structure of PHPT tests follows this basic pattern:
--TEST--
strtr() function - basic test for strstr()
--FILE--
/* Do not change this test it is a README.TESTING example. */
$trans = array("hello"=>"hi", "hi"=>"hello", "a"=>"A", "world"=>"planet");
var_dump(strtr("# hi all, I said hello world! #", $trans));
?>
--EXPECT--
string(32) "# hello All, I sAid hi planet! #"
As we can see the –TEST– section holds a one-line title for the test and should make it pretty clear as to what is being tested. The –FILE– section is the actual test code which will be internally converted into a .php file and executed. And finally the –EXPECT– section will be compared to the .php output to see if they are the same, var_dump is recommended for the tests. These are just the most standard sections – the full documentation can be checked on the QA site under phpt format,. You will come across situations where these blocks are not sufficient, but generally you will be checking if function X is returning value Y when passed parameter Z, so the above is usually all you need. One block is very important namely the –CREDIT– block. Putting your name here is very important so you can be reached in case of problems, and also to ensure that you get credit for writing tests for PHP. We used this format in our testfest:
--CREDIT-- Rafael Dohms <myemail@gmail.com> #PHPSPTestFest 2009-04-DD
Getting back to the code above, let’s try to write a test to cover the untested line we selected above. Let’s look closely at the parse function and figure out what we need to test:
zend_parse_parameters(ZEND_NUM_ARGS() TSRMLS_CC, "rllll", &IM, &red, &green, &blue, &alpha)
The important part to look at is the “rllll” as it defines the type needed for each of the parameters. The first parameter for example requires a Resource as stated by the “r”, whereas the rest of them need Long value. If we look at the PHP manual for this function we will get:
int imagecolorallocatealpha ( resource $image , int $red , int $green , int $blue , int $alpha )
It confirms what we saw in the code, but it might also show wrong information, so this would be another point where you can help PHP updating and fixing the documentation. Let’s write a test to make sure that the function sends an error message if we pass something that is not a Resource as its first parameter. Looking inside the ext/gd/tests folder we do not find any imagecolorallocatealpha files, so according to the naming conventions above we should use the name imagecolorallocatealpha_error1.phpt for our test file, since we are checking error conditions and it is the first error test for this function. (in sequence we can user error2, error3… for the other parameters)
--TEST-- Testing imagecolorallocatealpha(): Wrong types for parameter 1 --CREDITS-- Rafael Dohms <rdohms [at] gmail [dot] com>
The first step is to get the test title right and tag it with our credit so we can be found in case of future changes or problems. Make sure your title describes the objective without being an essay.
--SKIPIF--
<?php
if (!extension_loaded("gd")) die("skip GD not present");
?>
We did not mention the –SKIPIF– section before, but it is a key section in this case. This is very important because we are testing a function of the GD extension, so this means that depending on the local environment it might actually not be available, for example if GD is not enabled. In this case we do not want to execute these tests since they would all fail, leading to false negatives. This section can also be used for other skip criteria, like 32 bit Vs. 64 bit tests, or any other condition that you feel might affect your test. The section itself is very simple – just do a check and in case the conditions for skipping are met, call a die command and the test will be marked as SKIP not FAIL in the final execution.
--FILE--<?php $resource = tmpfile(); imagecolorallocatealpha($resource, 255, 255, 255, 50); imagecolorallocatealpha('string', 255, 255, 255, 50); imagecolorallocatealpha(array(), 255, 255, 255, 50); imagecolorallocatealpha(null, 255, 255, 255, 50); ?>
This is the body of the actual test – it is really simple and atomic. What we did is to call the function various times with different types of parameters. In the first call we are actually testing line 1859, because tmpfile returns a valid Resource, however not an Image Resource and line 1859 should catch that. The rest of the parameters receive valid values, in this case integers. Testing various types is interesting because you might find a specific type that causes non-expected behavior. Since we are hoping for error messages, we do not even need the return value of the function.
--EXPECTF-- Warning: imagecolorallocatealpha(): supplied resource is not a valid Image resource in %s on line %dWarning: imagecolorallocatealpha() expects parameter 1 to be resource, %s given in %s on line %dWarning: imagecolorallocatealpha() expects parameter 1 to be resource, array given in %s on line %dWarning: imagecolorallocatealpha() expects parameter 1 to be resource, null given in %s on line %d
Finally we now have a –EXPECTF– block which is the same as a –EXPECT– block but uses a printf-like structure to skip out on dynamic parts of the test. As we can see strings such as file name or line are replaced by %s or %d meaning it will fit any value. This is important because different environments might return different file paths. You can also use blocks with regular expressions, but this format is much more simple and easy to read. As we can see we are expecting an error message for each of the calls we made.
Tip: To get these error messages, execute the phpt file using a php executable, like: php imagecolorallocatealpha_error1.phpt. The file will be executed and everything outside <? tags will be output, and the actual output for the –FILE– block will be output as well, just copy-paste and replace dynamic bits.
Tip 2: the error message usually states the given type, as in: string given; If you are expecting the string type in the error message, replace it with %s to make your test compatible with PHP 5.3- and PHP6, since in this case it would print “unicode string” instead of “string”
4. Executing a test
Your test is ready for the drill, now we need the drillmaster to pick it up and make it jump through the hoops. We have seen that make test executes the PHP tests, but with 9000+ tests this is too time consuming to test one new test. So we should get it to execute only our test and the related tests we need. To do this we can use the TESTS parameter which will define which tests should be executed:
make test TESTS=ext/gd/tests/imagecolorallocatealpha_*.phpt
You can either pass it the full path to your test or use a wildcard (*) to execute all tests that match the pattern – this is useful if we decided to write tests for all parameters and want to run them all at once. This is the resulting output:
The test passes as we can see by the PASS text present before the test title (–TEST– block). Note that if this test had failed, the test framework would have created a few additional files to help you debug the problem. These files have the same name as the original .phpt but different extensions:
- .out – the output for the php code
- .php – source code generated for the test (–FILE–)
- .diff – a diff between –EXPECT– and the actual output
- .exp – the isolated text in the –EXPECT– section
- .log – a log of all information related (output, expected…)
These files are extremely useful in debugging. They are auto-removed once you get the test to PASS.
5. Submitting you test to PHP
There are a few ways to get your test into the PHP SVN. The easiest one is to participate in a TestFest event. All tests written at these events are submitted to a separate repository and then migrated to the official repository. The second way is to send your test file to the QA mailing list (use pastebin) and get someone to review it. The final way is a mix of the previous ones, participate in a TestFest event.. get a taste for tests, write a whole bunch of them, and at the end, apply for a SVN account. If you play your cards right and keep on writing tests you will easily get Karma and will become an official test committer in PHP. Hey it happened to me, true story, so it just depends on you and your willingness to write tests.
Conclusion
Now htat you have all the tools you need, what are you waiting for? Go help out the PHP Community. But don’t stop there, get the taste for tests and bring them to your application as well using PHPUnit and TDD – which I hope to write more about soon.
Related posts:
