Why were changes needed?

The WordPress test suite uses the industry standard PHPUnit tool to run the PHP unit tests.

Over the years, PHPUnit has seen a number of changes, new assertions being introduced, different annotations and more, but most notably, as of PHPUnit 8.0, a void return type was added to the typical fixture setUp() and tearDown() methods.

This void return type is problematic in the context of WordPress, as return types in general were only introduced in PHP 7.0 and the void return type wasn’t introduced until PHP 7.1.
While WordPress still has a minimum PHP version of PHP 5.6, the void return type can not be introduced in the test suite as it would inhibit the tests from being run on PHP 5.6 and 7.0.

At the same time, having to run the tests on older PHPUnit versions (PHPUnit < 8.0) made it increasingly difficult to get the test suite to run on new PHP versions, like PHP 8.0 and the upcoming PHP 8.1 (expected end of November) as these older PHPUnit versions are no longer supported and are not being made compatible with newer PHP versions anymore.

Over the past few years, a number of different solution directions were explored and rejected, largely due to the large maintenance burden these would add to the small team of WordPress contributors maintaining the test framework.

With the upcoming release of PHP 8.1 as a driver, we took another look at this problem and the available tooling in the wider PHP field and a solution has now been implemented which should future-proof the test suite for, at least, a number of years.

The solution

The implemented solution is based on the external PHPUnit Polyfills library, which “allows for creating PHPUnit cross-version compatible tests by offering a number of polyfills for functionality which was introduced, split up or renamed in PHPUnit”.

The PHPUnit Polyfills also solves the void conundrum via a tailored TestCase using snake_case methods.

In effect, this means that the WP Core test suite can now run on all PHPUnit versions between PHPUnit 5.7.21 up to the latest release (at the time of writing: PHPUnit 9.5.10), which allows for running the test suite against all supported PHP versions using the most appropriate PHPUnit version for that PHP version.

It also means that, as of mid August, the tests are being run against PHP 8.1 and fixes for PHP 8.1 compatibility are currently being made.

What has changed?

1. The Composer lock file has been removed.
   The version constraints in the composer.json file have been made stricter to ensure the developer experience is not negatively impacted by this with regards to coding standards checks.
2. The PHPUnit Polyfills library at version ^1.0.1 has been added as a Composer dev dependency.
3. All WordPress Core tests now use PHPUnit 9.x assertions and expectations.
4. All WordPress Core tests now use snake_case fixture methods, i.e. set_up() instead of setUp() and tear_down() instead of tearDown().
5. The minimum supported PHPUnit version has been raised to PHPUnit 5.7.21 (was 5.4.0).
6. The WordPress Core test bootstrap file will no longer throw an error when the tests are being run with PHPUnit 8.x or 9.x.
7. The WordPress Core test bootstrap file will throw an error when the PHPUnit Polyfills are not available or do not comply with the minimum version requirements.
8. All WP Core native assertions now have an extra, optional $message parameter, just like all PHPUnit native assertions.
   Please use this parameter in all tests which contain more than one assertion to make debugging tests easier.
9. The WP_UnitTestCase_Base::setExpectedException() method is deprecated and should no longer be used.
10. The WP_UnitTestCase_Base::checkRequirements() method is deprecated and no longer functional, and in reality hasn’t been for a long time for anyone using it in combination with PHPUnit 7.0+.
11. The copies of the PHPUnit SpeedTrapListener classes have been removed as they were never actively used in Core.
    Anyone who still wants to use the SpeedTrapListener can install it separately.
12. The copies of the PHPUnit 9.x MockObject classes which were introduced in the WP Core test suite in WP 5.6 have been removed, as they are no longer needed when the tests are run on the appropriate PHPUnit version for the PHP version used.

While the above changes have been made in WordPress 5.9, a minimal selection of these changes has been backported to WordPress 5.2 – 5.8:

1. The PHPUnit Polyfills at version ^1.0.1 is now a requirement for the test suites in WP 5.2 – 5.8 and this requirement will be enforced via the test bootstrap.
2. … which makes all the polyfills for PHPUnit 9.x assertions and expectations available when running tests against WP 5.2 – 5.8.
3. Additionally, snake_case wrapper methods have been added for the camelCase fixture method names, meaning that for WP 5.2 – 5.8, the snake_case fixture method names will work without needing further work-arounds, both for fixture declarations as well as for calling the parent::set_up() and the likes.
   
   There is one caveat to this: the backported implementation presumes a fixed order for calling the parent (camelCase) methods versus the child (snake_case) methods: for set_up*() methods: parent first, child second; for tear_down*() methods: child first, parent second.
   This is the standard order, but if you have a fixture method which diverges from this or doesn’t call the parent, you may get unexpected results.

These backports allow for backporting future (security) fixes for WordPress itself without having to make the accompanying tests compatible with older PHPUnit versions.

These backports will also make it more straightforward for extenders to continue to test their plugin/theme against multiple WordPress versions.

For full details about all the changes, please have a read through Trac ticket 46149.

Changes under the hood which should not be noticeable:

A new PHPUnit_Adapter_TestCase class has been added. This class is nested in-between the WP_UnitTestCase_Base class and the PHPUnit TestCase class and provides the PHPUnit cross-version adapter layer.

All PHPUnit assertion polyfill methods which previously existed in WP Core have been removed as they are no longer necessary now this functionality is provided via the PHPUnit Polyfills library.
All polyfills for assertions/expectations which were previously in WP are still available, they are now just provided via the Polyfills package.

As for the Docker set up: the PHPUnit container is no longer needed and has been removed from the docker-compose config.

What hasn’t changed:

• The PHPUnit class aliases (for support of PHPUnit 5), which WP provided are still available, though shouldn’t be needed anymore.
• You can still extend the WP_UnitTestCase class for your tests and will receive access to everything which was available before + more (i.e. a complete set of polyfills).

Future changes

There is a ticket open to rename some of the WordPress native test helper methods to handle the “doing it wrong” and WP native deprecation notices, as the current method names (too) closely resemble a PHPUnit native method name, which can easily lead to confusion and use of the wrong methods in tests.

When that ticket is actioned, this dev-note will be updated with the relevant information.

What does this mean for contributors to WordPress Core?

In general:

If you use Composer locally, please run composer update --with-all-dependencies (or composer update -W for short) from the root of your WordPress clone now to make sure your install is updated and to get the most appropriate versions of the dependencies for the PHP version you are running on.

▶ Go on, do that now. This dev-note will wait patiently for you to come back.

You will need to run this command semi-regularly in the future (whenever the composer.json file has been updated).

For running the Core tests:

If you usually run the Core tests via Docker using the npm run test:php command, you can continue to do so and all should still work as expected.

If you usually run the Core tests via a Composer installed version of PHPUnit, again, you can continue to do so and all should still work as expected as long as you followed the above instructions to run composer update -W first.

If you usually run the Core tests via a PHAR file, you either have to run composer update -W once in a while or you have to set up a clone of the PHPUnit Polyfills repo. For more information about this last option, please see the set up information in the handbook.
If you are running locally on PHP 7.2 or higher, you may want to download a more recent PHPUnit PHAR file (PHPUnit 8 or 9) to benefit from the advances which have been made in PHPUnit.

For writing tests for Core:

You can now use the full range of assertions as available in PHPUnit 9.5 in your tests. Please use the most appropriate assertion available.

Test fixture methods MUST use snake_case method names from now on as per the below table.

Old name	New name
setUpBeforeClass()	set_up_before_class()
setUp()	set_up()
assertPreConditions()	assert_pre_conditions()
assertPostConditions()	assert_post_conditions()
tearDown()	tear_down()
tearDownAfterClass()	tear_down_after_class()

The Make Core handbook page about writing tests has been updated with this information.
The page has also been enhanced with more handy tips and tricks, so please have a read through!

What does this mean for plugins/themes running integration tests based on the WP Core test suite?

It is a known fact that there are a lot of plugins/themes which use the WordPress Core test framework as a basis for their integration tests.

If your plugin/theme is one of them, these changes will impact you as well.

Step-by-step: how to make your test setup compatible with these changes and with higher PHPUnit versions:

1. Run your tests against PHP 7.4 with PHPUnit 7.x and WP 5.8.1 and make sure there are no pre-existing errors/failures.
2. Add PHPUnit Polyfills as a Composer require-dev dependency (or inherit it from WP).
3. If you add the Polyfills as a requirement and only support WP 5.9 and higher, remove the requirement for PHPUnit in favour of letting the Polyfills handle it. This will prevent potential future version constraint conflicts.
   • If you still need/want to run your tests against older WP versions, keep the PHPUnit requirement and make sure it is set to ^5.7.21 || ^6.5 || ^7.5 and let CI (continuous integration script) handle removing that requirement for WP 5.9.
   • Or do it in reverse and remove the requirement for dev and add it back in CI for older WP versions.
4. Make sure the Polyfills autoloader is wired in to your test bootstrap.
   • If you’ve chosen to “inherit the Polyfills from WP”, in this context that means that you use a full clone of WordPress and will install the Composer dependencies for WordPress before running the tests. In that case, you should be all set.
   • If you use only a partial clone of WordPress, like when your tests have been set up using the WP-CLI scaffold command, or if you don’t run WordPress’ Composer setup, please make sure you load the Polyfills autoloader in your test bootstrap before running the WP native test bootstrap.
     • If you include your Composer vendor/autoload.php file as your test bootstrap before you run the WP native test bootstrap, you’re all set already, the Polyfills autoloader will be included automatically.
     • Alternatively, you can add a require_once 'path/to/vendor/yoast/phpunit-polyfills/phpunitpolyfills-autoload.php'; in your test bootstrap before including the WP native test bootstrap.
     • As a last alternative, you can declare a WP_TESTS_PHPUNIT_POLYFILLS_PATH constant containing the absolute path to the root directory of the PHPUnit Polyfills installation in your plugin/theme’s own test bootstrap file.
       Again, this constant must be declared prior to running the WP native test bootstrap file.
5. Search your codebase for declarations of the fixture methods, as well as calls to (parent) fixture methods, and replace camelCase with snake_case in the method names.
   Example:
   

   // Old:
   public function setUp() {
        parent::setUp();
        // Do something.
   }
   
   // New:
   public function set_up() {
       parent::set_up();
       // Do something.
   }

6. Verify your tests run without errors after the changes by running them against PHP 7.4 on PHPUnit 7.x with WP 5.8 .
7. Verify your tests run without errors after the changes by running them against PHP 7.4 on PHPUnit 7.x with WP trunk (= WP 5.9).
8. While using WP trunk/5.9, switch to PHPUnit 8.x – look out for deprecation notices PHPUnit throws and just literally do what they tell you to do.
9. While still using WP trunk/5.9, switch to PHPUnit 9.x – look out for deprecation notices PHPUnit throws and just literally do what they tell you to do.

Once you’ve run through these steps, your tests should be cross-version compatible with PHPUnit 5.7.21 – 9.5, able to run against the WordPress 5.2 to 5.9 branches and able to run on PHP 5.6 – 8.1.

Next, you may want to run your tests against PHP 8.0 and 8.1 using PHPUnit 9.x with WP 5.9 to see if your plugin/theme is compatible with these more recent PHP versions.

What to do when running tests in CI against multiple WP/PHP combinations?

If you are running your plugin/theme integration tests against multiple WordPress and PHP combinations, you will most likely need to make some adjustments to your Continuous Integration (CI) script(s).

Which exact changes you need to make depends entirely on your specific setup. There is no “one size fits all” solution.

As a general rule of thumb:

• WP 5.2 – 5.5 is able to run tests against PHP 5.6 – 7.4 with PHPUnit 5.x (PHP 5.6 and 7.0) – 7.x (PHP 7.1 and higher).
• WP 5.6 – 5.8 is able to run tests against PHP 5.6 – 8.0 with PHPUnit 5.x (PHP 5.6 and 7.0) – 7.x (PHP 7.1 and higher).
• WP 5.9 and higher is able to run tests against PHP 5.6 – 8.1 with PHPUnit 5.x – 9.x (use the most appropriate PHPUnit version for each PHP version).

Also see the PHP Compatibility and WordPress Versions and PHPUnit Compatibility and WordPress Versions pages in the Make Core handbook.

Other typical things to take into account and to work around when needed:

• Is there a config - platform - php setting in your composer.json which fixes the PHP version to a specific version – typically PHP 5.6 – for installing dependencies ?
  If so, you may need to either selectively remove this setting or run Composer with --ignore-platform-reqs for certain WP/PHP combinations in your test matrix.
• Has the composer.lock file been committed ?
  In that case, you may need to either selectively remove that file in CI before running composer install; or run composer update -W for certain WP/PHP combinations in your test matrix.
• Do you use a complete clone of WP ?
  For WP 5.2 – 5.8, you’ll need to install the WP dependencies by using composer install.
  For WP 5.9 and higher, you’ll need to install the WP dependencies by using composer update -W.

To make sure you run the test against the right PHPUnit version, you may need to run (a variation on):

Anticipating some frequently asked questions

I’m a plugin/theme maintainer, but don’t use Composer, can I still run my integration tests?

Yes, but you do need to make sure that either the Polyfills are available via a Composer global or local install or via some other manner, like a clone of the repo.

If you haven’t looked at Composer before, now might be a good time to take a look at it.

I’m running my tests via another tool stack (like BrainMonkey, WP Mock, PHP Mock, WP Browser, PestPHP), how do the changes made to the WordPress test suite affect me?

Short answer: They don’t.

Long answer: if you want to run your tests against multiple PHP and PHPUnit combinations, you may still find the PHPUnit Polyfills library helpful to you.

If you’ve not heard of the above mentioned tools before and want to read up on them, here are some links:

• BrainMonkey
• WP Mock
• PHP Mock
• WP Browser
• PestPHP

I used the WP-CLI scaffold command to set up my integration tests. How do the changes made to the WordPress test suite affect me?

There is no automated way right now to adapt existing tests for which the initial was created via the WP-CLI scaffold command, to make use of the new setup.

The current recommendation is to go through the steps in “What does this mean for plugins/themes running integration tests based on the WP Core test suite?“, which might be more or less involved depending on what version of the scaffolded test setup you are currently using.

A future version of the WP-CLI scaffold plugin-tests command will provide an upgrade mechanism to automatically upgrade an existing test setup to the new requirements. This will include adding a fully Composer-based testing setup as a replacement for the current bootstrap logic, making a composer update possible in the future to keep up with further test setup changes.

If you’re interested in learning more about these plans for the future, please subscribe to the issue on GitHub to stay informed.

I’m using WP Test Utils for my unit and integration tests. How do the changes made to the WordPress test suite affect me?

WP Test Utils is a library offering utilities for both unit testing and integration testing for WordPress plugins and themes. WP Test Utils already includes the PHPUnit Polyfills.

For the unit testing part, which is based on BrainMonkey, you are not affected by the changes.

If you use the integration testing utilities, you will need to make the change from camelCase to snake_case for the fixture methods in your test suite and you can now potentially widen the PHPUnit version requirements for your integration tests (also see the information about this in step 3 of the “Step by step” guide and the information about adjusting CI scripts).

Presuming you were already using the PHPUnit Polyfills provided by the Test Utils to use modern assertions, that’s it. You’re done.

WP Test Utils will continue to handle the integration test bootstrapping, which allows for running the tests against multiple WordPress and PHP versions.

The first version of WP Test Utils which has full support for the test framework changes made in WP 5.9, is WP Test Utils 1.0.0.

WP Test Utils 1.0.0 also includes improved support for integration tests which were created using the WP-CLI scaffold command and support for running tests against WP versions which don’t include the backports, like WP 5.2 – 5.8 point releases released before today, as well as WP < 5.2.

---

Props to @hellofromtonya, @johnbillion, @dingo_d, @netweb, @sergeybiryukov, @swissspidy, @schlessera for reviewing ahead of publication.

#5-9, #build-test-tools, #dev-notes, #phpunit, #unit-tests