Skip to content

Acceptance Testing

Running Acceptance Tests on Magento 2

Magento 2 has a full suite of acceptance tests built into it, which covers everything from critical paths to regressions and bug fixes. When developing your own Magento 2 site you should be making use of these, as well as building your own tests to cover bespoke functionality.

Getting these setup and running them for the first time is a bit complicated, so the process that we use is documented below.

Configuring the project

Magento provides a guide for how to do this here, these are the key steps.

Setup the project

From the root of the project run the following command

vendor/bin/mftf build:project

This will run the initial setup for the testing framework

Update the .env file

You will need to edit the following file dev/tests/acceptance/.env

Set the MAGENTO_BASE_URL to match the URL of your local development instance. If you are running the tests in a container you will also need to update the hosts file of that container to include this

!! Important Make sure this uses the https version of the base URL if you have configured this, otherwise you will hit issues with secure URLs

Set the other variables as so


These values will be used when we set up the testing database

Start selenium

The tests need selenium to be running, so start this before continuing.

Create a testing database

We are going to install a fresh version of magento and configure it. This means that we need to create a database and user before we continue.

Create the test suite file

We are going to create a suite of tests to run against the site. To do this we need to create an XML file with the details of it.

In the dev/tests/acceptance/tests/_suite/ directory create a file called ClientName.xml

To start with we will just run the checkout tests, however this can be extended to include any set of tests that you want. See here for full details on how to do this.

For the moment, put the following in the file

<?xml version="1.0" encoding="UTF-8"?>
<suites xmlns:xsi="" xsi:noNamespaceSchemaLocation="../../vendor/magento/magento2-functional-testing-framework/src/Magento/FunctionalTestingFramework/Suite/etc/suiteSchema.xsd">
    <suite name="clientTests">
            <group name="checkout" />

The name value in the suite tag is what the tests are called and will be used later on in the test runner.

Create the test runner script

To run the tests the following will be done

  • Caches and Sessions will be cleared
  • A clean version of Magento will be installed into a test database
  • That database will be configured correctly to allow the tests to run
  • It will also have client specific configuration setup
  • The site will be put into production mode to match the live version
  • A test suite will be generated and run

This can all be done using the script below

#!/usr/bin/env bash
readonly DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )";
cd ${DIR};
set -e
set -u
set -o pipefail

# -------------------------------------- #
# Here we set the variables for the site #
# -------------------------------------- #
# Magento Root Directory
# Mage run location
# URL of the site - no protocol and no trailing slash
# Database details
# Theme ID - Run `n98-magerun2.phar dev:theme:list` to find this
# Name of the test suite to run - See

# --------------------- #
# End of Variable setup #
# --------------------- #

echo "
Pre run sanity checks

if [[ ! -f ${MAGENTO_ROOT_DIR}/dev/tests/acceptance/.env ]]
echo "
Could not find the .env in ${MAGENTO_ROOT_DIR}/dev/tests/acceptance/ !
Have you set up MFTF?

exit 5

CLI_SETUP=`curl -o /dev/null -k -w "%{http_code}" -s https://${BASE_URL}/dev/tests/acceptance/utils/command.php`

if [[ ${CLI_SETUP} == 404 ]]
   echo "
Web based CLI is not setup! Follow the instructions here

   exit 5

echo "
Going to clear out the caches and sessions before continuing
rm -rf ${MAGENTO_ROOT_DIR}/var/cache/*
rm -rf ${MAGENTO_ROOT_DIR}/var/sessions/*

echo "
Going to install a clean version of Magento
php ${MAGENTO_ROOT_DIR}/bin/magento setup:install -q \
    --language="en_US" \
    --timezone="UTC" \
    --currency="USD" \
    --base-url="http://${BASE_URL}/" \
    --base-url-secure="https://${BASE_URL}/" \
    --admin-firstname="John" \
    --admin-lastname="Doe" \
    --backend-frontname="admin" \
    --admin-email="" \
    --admin-user="admin" \
    --use-rewrites=1 \
    --admin-use-security-key=0 \
    --admin-password="123123q" \
    --db-name="${DB_NAME}" \
    --db-user="${DB_USER}" \
    --db-password=${DB_PASS} \

echo "
Going to set the required configuration
php ${MAGENTO_ROOT_DIR}/bin/magento config:set web/secure/use_in_adminhtml 1
php ${MAGENTO_ROOT_DIR}/bin/magento config:set admin/security/admin_account_sharing 1
php ${MAGENTO_ROOT_DIR}/bin/magento config:set cms/wysiwyg/enabled disabled

echo "
Using the correct theme
php ${MAGERUN_LOCATION} db:query \
"INSERT INTO core_config_data (scope,scope_id,path,value) VALUES ('default', 0, 'design/theme/theme_id', ${THEME_ID})"

# Ideally required configuration should be included in setup scripts so it is applied automatically when magento is installed
# If this is not the case, then you can add any other configuration below

echo "
Setting the site into production mode
php ${MAGENTO_ROOT_DIR}/bin/magento deploy:mode:set production

echo "
Generating and then running the test suite
php ${MAGENTO_ROOT_DIR}/vendor/bin/mftf run:group ${TEST_SUITE} -r

To use this create a copy of it in the root of the project. Then do the following

  • Marke sure Magerun is installed
  • Update the variables at the top of the file
    • Make sure the database details are for the testing database, not the real one
    • Fetch the theme ID using the magerun command
    • Update the test suite with the details we created earlier
  • If there is any custom configuration that is not set using setup scripts and then it below the section where the theme is set
  • Make a copy of the current app/etc/env.php file as it will be overwritten
  • Run the file

The file will carry out a couple of sanity checks and then install a clean version of Magento and run the tests.

These can take some time to run through, once it they are complete you will have a report of which passed and which failed.

Fixing failing tests

Some of the tests will fail due to differences between the Luma theme and the Theme the client is using. To fix these you will need to create overrides for the tests.

First you will need to create a new testing module. Once that is done you will need to update / override the tests or selectors.

Magento provides a lot of documentation on how to do this here

General Help

These are some tips I've found useful when working with this

Running Individual Tests

The testing framework is built on top of Codeception, to run individual tests we need to use the codecept executable. The acceptance test uses it's own sub suite of functional tests that are separate from the main `functional tests directory.

To run an individual first cd to your acceptance test directory.

php ../../../vendor/bin/codecept run \
 --steps functional tests/functional/Magento/FunctionalTest/_generated/clientTest/OnePageCheckoutWithAllProductTypesTestCest.php

The first argument is the suite name functional, this tells Codeception which yaml config to use although in the case of MFTF there will only ever be one suite.

Editing tests

The generate command combines all of the XML files and creates Codeception tests to run.

These are created in the dev/tests/acceptance/tests/functional/Magento/FunctionalTest/_generated/${suiteName} directory.

It is possbile to edit these directly and then run them using the following command

php vendor/bin/mftf run:group ${TEST_SUITE} -k

This runs the tests without regenerating them. Once you have checked what you needed to make sure to run the tests after generating them again.

Debugging calls

One of the benefits of doing this, is it allows you to set an xDebug cookie when the browser is running.

This is done by adding the following line to the test


If running this from the CLI you will want to make sure the you are not listening for connections when running the test command and start before the browser makes the request.

This is because the command makes calls which can get picked up by PHPStorm before the browser is launched. On the otherhand if you want to step through the actual test this can also be done.

Running the tests with Zalenium

If you are running the tests with Zalenium you will need to add "edmondscommerce/magento2-zalenium" to you composer dev dependencies to get better Zalenium output.

You will then need to add this class to the funtional.suite.yml similar to below.

        # ...
        - \EdmondsCommerce\Zalenium\Zalenium


To install Allure you'll need to run this command (assuming npm is already installed)

sudo npm install -g allure-commandline --save-dev

In order to run Allure, you'll also need to install java

sudo bash
yum install java
After running the tests, run this command in the root of the project to see the report in the browser. You'll need to open the generated URL
allure serve dev/tests/acceptance/tests/_output/allure-results/

Overriding test steps

  • see if your step is in an action group or not
  • if the step is in an action group, then you'll need to override the step in the action group
  • if the step is in a test by itself, then you'll need to override the step in the test itself
  • create a new M2 module for example app/code/EdmondsCommerce/TestOverrides
  • place your override files in here app/code/EdmondsCommerce/TestOverrides/Test/Mftf in their appropriate folders to replicate the original path For example, to override an action group use something like this
  • regenerate the tests with vendor/bin/mftf generate:tests
  • check your test has the change

Magento provides a lot of documentation on how to manipulate test steps here

Identifying the step to be overridden

Once we know what needs to be overridden we should locate the step key from the test, this is usually in a comment next to the step.

$I->click("//div[@id='account-nav']//a[text()='Address Book']"); // stepKey: goToAddressBookGoToAddressBookPage

Search for the stepKey in vendor to find the action group, section or test to override. We should override as high up the chain as possible to fix the problem across multiple tests.

Codeception Config

When using a newer version of the chrome driver you will need to change the options passed to selenium as the previous way is deprecated.

To do this we will need to change the functional.suite.yml file.

The options for chrome should look like this:

        args: ["--window-size=1280,1024", "--disable-extensions", "--enable-automation", "--enable-javascript", "--enable-Passthrough"]