Acceptance Testing

Running Acceptance Tests on Magento 2

Magento 2 has a full suite of acceptance tests built into it, which covers everything from critical paths to regressions and bug fixes. When developing your own Magento 2 site you should be making use of these, as well as building your own tests to cover bespoke functionality.

Getting these setup and running them for the first time is a bit complicated, so the process that we use is documented below.

Configuring the project

Magento provides a guide for how to do this here, these are the key steps.

Setup the project

From the root of the project run the following command

1
vendor/bin/mftf build:project

This will run the initial setup for the testing framework

Update the .env file

You will need to edit the following file dev/tests/acceptance/.env

Set the MAGENTO_BASE_URL to match the URL of your local development instance. If you are running the tests in a container you will also need to update the hosts file of that container to include this

!! Important Make sure this uses the https version of the base URL if you have configured this, otherwise you will hit issues with secure URLs

Set the other variables as so

1
2
3
MAGENTO_BACKEND_NAME=admin
MAGENTO_ADMIN_USERNAME=admin
MAGENTO_ADMIN_PASSWORD=123123q

These values will be used when we set up the testing database

Start selenium

The tests need selenium to be running, so start this before continuing.

Create a testing database

We are going to install a fresh version of magento and configure it. This means that we need to create a database and user before we continue.

Create the test suite file

We are going to create a suite of tests to run against the site. To do this we need to create an XML file with the details of it.

In the dev/tests/acceptance/tests/_suite/ directory create a file called ClientName.xml

To start with we will just run the checkout tests, however this can be extended to include any set of tests that you want. See here for full details on how to do this.

For the moment, put the following in the file

1
2
3
4
5
6
7
8
<?xml version="1.0" encoding="UTF-8"?>
<suites xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="../../vendor/magento/magento2-functional-testing-framework/src/Magento/FunctionalTestingFramework/Suite/etc/suiteSchema.xsd">
    <suite name="clientTests">
        <include>
            <group name="checkout" />
        </include>
    </suite>
</suites>

The name value in the suite tag is what the tests are called and will be used later on in the test runner.

Create the test runner script

To run the tests the following will be done

  • Caches and Sessions will be cleared
  • A clean version of Magento will be installed into a test database
  • That database will be configured correctly to allow the tests to run
  • It will also have client specific configuration setup
  • The site will be put into production mode to match the live version
  • A test suite will be generated and run

This can all be done using the script below

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
#!/usr/bin/env bash
readonly DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )";
cd ${DIR};
set -e
set -u
set -o pipefail
standardIFS="$IFS"
IFS=$'\n\t'

# -------------------------------------- #
# Here we set the variables for the site #
# -------------------------------------- #
# Magento Root Directory
MAGENTO_ROOT_DIR=${DIR}
# Mage run location
MAGERUN_LOCATION="${DIR}/n98-magerun2.phar"
# URL of the site - no protocol and no trailing slash
BASE_URL="www.example.com"
# Database details
DB_NAME="magento_testi_database"
DB_USER="db_user"
DB_PASS="password123"
# Theme ID - Run `n98-magerun2.phar dev:theme:list` to find this
THEME_ID=4
# Name of the test suite to run - See https://devdocs.magento.com/mftf/docs/suite.html
TEST_SUITE="clientTest"

# --------------------- #
# End of Variable setup #
# --------------------- #


echo "
---------------------
Pre run sanity checks
---------------------
"

if [[ ! -f ${MAGENTO_ROOT_DIR}/dev/tests/acceptance/.env ]]
then
echo "
Could not find the .env in ${MAGENTO_ROOT_DIR}/dev/tests/acceptance/ !
Have you set up MFTF?
https://devdocs.magento.com/mftf/docs/getting-started.html
"

exit 5
fi

CLI_SETUP=`curl -o /dev/null -k -w "%{http_code}" -s https://${BASE_URL}/dev/tests/acceptance/utils/command.php`

if [[ ${CLI_SETUP} == 404 ]]
then
   echo "
Web based CLI is not setup! Follow the instructions here
https://devdocs.magento.com/mftf/docs/getting-started.html#nginx-settings
"

   exit 5
fi

echo "
------------------------------------------------------------
Going to clear out the caches and sessions before continuing
------------------------------------------------------------
"
rm -rf ${MAGENTO_ROOT_DIR}/var/cache/*
rm -rf ${MAGENTO_ROOT_DIR}/var/sessions/*

echo "
-------------------------------------------
Going to install a clean version of Magento
-------------------------------------------
"
php ${MAGENTO_ROOT_DIR}/bin/magento setup:install -q \
    --language="en_US" \
    --timezone="UTC" \
    --currency="USD" \
    --base-url="http://${BASE_URL}/" \
    --base-url-secure="https://${BASE_URL}/" \
    --admin-firstname="John" \
    --admin-lastname="Doe" \
    --backend-frontname="admin" \
    --admin-email="admin@example.com" \
    --admin-user="admin" \
    --use-rewrites=1 \
    --admin-use-security-key=0 \
    --admin-password="123123q" \
    --db-name="${DB_NAME}" \
    --db-user="${DB_USER}" \
    --db-password=${DB_PASS} \
    --cleanup-database

echo "
---------------------------------------
Going to set the required configuration
---------------------------------------
"
php ${MAGENTO_ROOT_DIR}/bin/magento config:set web/secure/use_in_adminhtml 1
php ${MAGENTO_ROOT_DIR}/bin/magento config:set admin/security/admin_account_sharing 1
php ${MAGENTO_ROOT_DIR}/bin/magento config:set cms/wysiwyg/enabled disabled

echo "
-----------------------
Using the correct theme
-----------------------
"
php ${MAGERUN_LOCATION} db:query \
"INSERT INTO core_config_data (scope,scope_id,path,value) VALUES ('default', 0, 'design/theme/theme_id', ${THEME_ID})"

#
# Ideally required configuration should be included in setup scripts so it is applied automatically when magento is installed
# If this is not the case, then you can add any other configuration below
#

echo "
-------------------------------------
Setting the site into production mode
-------------------------------------
"
php ${MAGENTO_ROOT_DIR}/bin/magento deploy:mode:set production

echo "
------------------------------------------
Generating and then running the test suite
------------------------------------------
"
php ${MAGENTO_ROOT_DIR}/vendor/bin/mftf run:group ${TEST_SUITE} -r

To use this create a copy of it in the root of the project. Then do the following

  • Marke sure Magerun is installed
  • Update the variables at the top of the file
    • Make sure the database details are for the testing database, not the real one
    • Fetch the theme ID using the magerun command
    • Update the test suite with the details we created earlier
  • If there is any custom configuration that is not set using setup scripts and then it below the section where the theme is set
  • Make a copy of the current app/etc/env.php file as it will be overwritten
  • Run the file

The file will carry out a couple of sanity checks and then install a clean version of Magento and run the tests.

These can take some time to run through, once it they are complete you will have a report of which passed and which failed.

Fixing failing tests

Some of the tests will fail due to differences between the Luma theme and the Theme the client is using. To fix these you will need to create overrides for the tests.

First you will need to create a new testing module. Once that is done you will need to update / override the tests or selectors.

Magento provides a lot of documentation on how to do this here

General Help

These are some tips I've found useful when working with this

Editing tests

The generate command combines all of the XML files and creates Codeception tests to run.

These are created in the dev/tests/acceptance/tests/functional/Magento/FunctionalTest/_generated/${suiteName} directory.

It is possbile to edit these directly and then run them using the following command

1
php vendor/bin/mftf run:group ${TEST_SUITE} -k

This runs the tests without regenerating them. Once you have checked what you needed to make sure to run the tests after generating them again.

Debugging calls

One of the benefits of doing this, is it allows you to set an xDebug cookie when the browser is running.

This is done by adding the following line to the test

1
$I->setCookie('XDEBUG_SESSION', 'XDEBUG_ECLIPSE');

If running this from the CLI you will want to make sure the you are not listening for connections when running the test command and start before the browser makes the request.

This is because the command makes calls which can get picked up by PHPStorm before the browser is launched. On the otherhand if you want to step through the actual test this can also be done.