Acceptance Testing

Running Acceptance Tests on Magento 2¶

Magento 2 has a full suite of acceptance tests built into it, which covers everything from critical paths to regressions and bug fixes. When developing your own Magento 2 site you should be making use of these, as well as building your own tests to cover bespoke functionality.

Getting these setup and running them for the first time is a bit complicated, so the process that we use is documented below.

Configuring the project¶

Magento provides a guide for how to do this here, these are the key steps.

Setup the project¶

From the root of the project run the following command

vendor/bin/mftf build:project

This will run the initial setup for the testing framework

Update the .env file¶

You will need to edit the following file dev/tests/acceptance/.env

Set the MAGENTO_BASE_URL to match the URL of your local development instance. If you are running the tests in a container you will also need to update the hosts file of that container to include this

!! Important Make sure this uses the https version of the base URL if you have configured this, otherwise you will hit issues with secure URLs

Set the other variables as so

MAGENTO_BACKEND_NAME=admin
MAGENTO_ADMIN_USERNAME=admin
MAGENTO_ADMIN_PASSWORD=123123q

These values will be used when we set up the testing database

Start selenium¶

The tests need selenium to be running, so start this before continuing.

Create a testing database¶

We are going to install a fresh version of magento and configure it. This means that we need to create a database and user before we continue.

Create the test suite file¶

We are going to create a suite of tests to run against the site. To do this we need to create an XML file with the details of it.

In the dev/tests/acceptance/tests/_suite/ directory create a file called ClientName.xml

To start with we will just run the checkout tests, however this can be extended to include any set of tests that you want. See here for full details on how to do this.

For the moment, put the following in the file

<?xml version="1.0" encoding="UTF-8"?>
<suites xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="../../vendor/magento/magento2-functional-testing-framework/src/Magento/FunctionalTestingFramework/Suite/etc/suiteSchema.xsd">
    <suite name="clientTests">
        <include>
            <group name="checkout" />
        </include>
    </suite>
</suites>

The name value in the suite tag is what the tests are called and will be used later on in the test runner.

Create the test runner script¶

To run the tests the following will be done

Caches and Sessions will be cleared
A clean version of Magento will be installed into a test database
That database will be configured correctly to allow the tests to run
It will also have client specific configuration setup
The site will be put into production mode to match the live version
A test suite will be generated and run

This can all be done using the script below

#!/usr/bin/env bash
readonly DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )";
cd ${DIR};
set -e
set -u
set -o pipefail
standardIFS="$IFS"
IFS=$'\n\t'

# -------------------------------------- #
# Here we set the variables for the site #
# -------------------------------------- #
# Magento Root Directory
MAGENTO_ROOT_DIR=${DIR}
# Mage run location
MAGERUN_LOCATION="${DIR}/n98-magerun2.phar"
# URL of the site - no protocol and no trailing slash
BASE_URL="www.example.com"
# Database details
DB_NAME="magento_testi_database"
DB_USER="db_user"
DB_PASS="password123"
# Theme ID - Run `n98-magerun2.phar dev:theme:list` to find this
THEME_ID=4
# Name of the test suite to run - See https://devdocs.magento.com/mftf/docs/suite.html
TEST_SUITE="clientTest"

# --------------------- #
# End of Variable setup #
# --------------------- #


echo "
---------------------
Pre run sanity checks
---------------------
"

if [[ ! -f ${MAGENTO_ROOT_DIR}/dev/tests/acceptance/.env ]]
then
echo "
Could not find the .env in ${MAGENTO_ROOT_DIR}/dev/tests/acceptance/ !
Have you set up MFTF?
https://devdocs.magento.com/mftf/docs/getting-started.html
"

exit 5
fi

CLI_SETUP=`curl -o /dev/null -k -w "%{http_code}" -s https://${BASE_URL}/dev/tests/acceptance/utils/command.php`

if [[ ${CLI_SETUP} == 404 ]]
then
   echo "
Web based CLI is not setup! Follow the instructions here
https://devdocs.magento.com/mftf/docs/getting-started.html#nginx-settings
"

   exit 5
fi

echo "
------------------------------------------------------------
Going to clear out the caches and sessions before continuing
------------------------------------------------------------
"
rm -rf ${MAGENTO_ROOT_DIR}/var/cache/*
rm -rf ${MAGENTO_ROOT_DIR}/var/sessions/*

echo "
-------------------------------------------
Going to install a clean version of Magento
-------------------------------------------
"
php ${MAGENTO_ROOT_DIR}/bin/magento setup:install -q \
    --language="en_US" \
    --timezone="UTC" \
    --currency="USD" \
    --base-url="http://${BASE_URL}/" \
    --base-url-secure="https://${BASE_URL}/" \
    --admin-firstname="John" \
    --admin-lastname="Doe" \
    --backend-frontname="admin" \
    --admin-email="admin@example.com" \
    --admin-user="admin" \
    --use-rewrites=1 \
    --admin-use-security-key=0 \
    --admin-password="123123q" \
    --db-name="${DB_NAME}" \
    --db-user="${DB_USER}" \
    --db-password=${DB_PASS} \
    --cleanup-database

echo "
---------------------------------------
Going to set the required configuration
---------------------------------------
"
php ${MAGENTO_ROOT_DIR}/bin/magento config:set web/secure/use_in_adminhtml 1
php ${MAGENTO_ROOT_DIR}/bin/magento config:set admin/security/admin_account_sharing 1
php ${MAGENTO_ROOT_DIR}/bin/magento config:set cms/wysiwyg/enabled disabled

echo "
-----------------------
Using the correct theme
-----------------------
"
php ${MAGERUN_LOCATION} db:query \
"INSERT INTO core_config_data (scope,scope_id,path,value) VALUES ('default', 0, 'design/theme/theme_id', ${THEME_ID})"

#
# Ideally required configuration should be included in setup scripts so it is applied automatically when magento is installed
# If this is not the case, then you can add any other configuration below
#

echo "
-------------------------------------
Setting the site into production mode
-------------------------------------
"
php ${MAGENTO_ROOT_DIR}/bin/magento deploy:mode:set production

echo "
------------------------------------------
Generating and then running the test suite
------------------------------------------
"
php ${MAGENTO_ROOT_DIR}/vendor/bin/mftf run:group ${TEST_SUITE} -r

To use this create a copy of it in the root of the project. Then do the following

Marke sure Magerun is installed
Update the variables at the top of the file
- Make sure the database details are for the testing database, not the real one
- Fetch the theme ID using the magerun command
- Update the test suite with the details we created earlier
If there is any custom configuration that is not set using setup scripts and then it below the section where the theme is set
Make a copy of the current app/etc/env.php file as it will be overwritten
Run the file

The file will carry out a couple of sanity checks and then install a clean version of Magento and run the tests.

These can take some time to run through, once it they are complete you will have a report of which passed and which failed.

Fixing failing tests¶

Some of the tests will fail due to differences between the Luma theme and the Theme the client is using. To fix these you will need to create overrides for the tests.

First you will need to create a new testing module. Once that is done you will need to update / override the tests or selectors.

Magento provides a lot of documentation on how to do this here

General Help¶

These are some tips I've found useful when working with this

Running Individual Tests¶

The testing framework is built on top of Codeception, to run individual tests we need to use the codecept executable. The acceptance test uses it's own sub suite of functional tests that are separate from the main `functional tests directory.

To run an individual first cd to your acceptance test directory.

php ../../../vendor/bin/codecept run \
 --steps functional tests/functional/Magento/FunctionalTest/_generated/clientTest/OnePageCheckoutWithAllProductTypesTestCest.php

The first argument is the suite name functional, this tells Codeception which yaml config to use although in the case of MFTF there will only ever be one suite.

Editing tests¶

The generate command combines all of the XML files and creates Codeception tests to run.

These are created in the dev/tests/acceptance/tests/functional/Magento/FunctionalTest/_generated/${suiteName} directory.

It is possbile to edit these directly and then run them using the following command

php vendor/bin/mftf run:group ${TEST_SUITE} -k

This runs the tests without regenerating them. Once you have checked what you needed to make sure to run the tests after generating them again.

Debugging calls¶

One of the benefits of doing this, is it allows you to set an xDebug cookie when the browser is running.

This is done by adding the following line to the test

$I->setCookie('XDEBUG_SESSION', 'XDEBUG_ECLIPSE');

If running this from the CLI you will want to make sure the you are not listening for connections when running the test command and start before the browser makes the request.

This is because the command makes calls which can get picked up by PHPStorm before the browser is launched. On the otherhand if you want to step through the actual test this can also be done.

Running the tests with Zalenium¶

If you are running the tests with Zalenium you will need to add "edmondscommerce/magento2-zalenium" to you composer dev dependencies to get better Zalenium output.

You will then need to add this class to the funtional.suite.yml similar to below.

modules:
    enabled:
        # ...
        - \EdmondsCommerce\Zalenium\Zalenium

Allure¶

To install Allure you'll need to run this command (assuming npm is already installed)

sudo npm install -g allure-commandline --save-dev

In order to run Allure, you'll also need to install java

sudo bash
yum install java

After running the tests, run this command in the root of the project to see the report in the browser. You'll need to open the generated URL

allure serve dev/tests/acceptance/tests/_output/allure-results/

Overriding test steps¶

see if your step is in an action group or not
if the step is in an action group, then you'll need to override the step in the action group
if the step is in a test by itself, then you'll need to override the step in the test itself
create a new M2 module for example app/code/EdmondsCommerce/TestOverrides
place your override files in here app/code/EdmondsCommerce/TestOverrides/Test/Mftf in their appropriate folders to replicate the original path For example, to override an action group use something like this
```
/home/catalin/Projects/owd-magento2/app/code/EdmondsCommerce/TestOverrides/Test/Mftf/ActionGroup/AssertStorefrontMiniCartItemsActionGroup.xml
```
regenerate the tests with vendor/bin/mftf generate:tests
check your test has the change

Magento provides a lot of documentation on how to manipulate test steps here

Identifying the step to be overridden¶

Once we know what needs to be overridden we should locate the step key from the test, this is usually in a comment next to the step.

$I->click("//div[@id='account-nav']//a[text()='Address Book']"); // stepKey: goToAddressBookGoToAddressBookPage

Search for the stepKey in vendor to find the action group, section or test to override. We should override as high up the chain as possible to fix the problem across multiple tests.

Codeception Config¶

When using a newer version of the chrome driver you will need to change the options passed to selenium as the previous way is deprecated.

To do this we will need to change the functional.suite.yml file.

The options for chrome should look like this:

capabilities:
    "goog:chromeOptions":
        args: ["--window-size=1280,1024", "--disable-extensions", "--enable-automation", "--enable-javascript", "--enable-Passthrough"]