Skip to content

Behat

Summary of Behat

The Behat tests simulate a users experience while having access to multiple features under the hood for detecting errors on the site. Our Behat tests also perform W3C validation and detect JavaScript errors on each page visited by the test suite. With more time invested into the tests detecting bugs and minimising errors is drastically increased before deployment.

The below options are examples of the structure of the way we test websites prioritising basic functionality first.

Basic Functionality (Desktop, Mobile) to test:

  • Homepage
  • Checkout
  • Product Page
  • Customer Account
  • Cart
  • Category Page
  • Navigation
  • Basic Search

Core Functionality (Desktop, Mobile, Tablet) to test:

  • Contact Forms
  • Searching
  • Mini Cart
  • Advanced Product Page Options
  • Delivery Options
  • Advanced Search

Advanced Functionality (Desktop, Mobile, Tablet) to test:

This will include bespoke tests catered towards website ie custom plugins and tests requested.

  • Admin
  • Custom Plugins
  • Performance

See an example video below of a test website with some of the feature tests from the basic functionality:

Finding and updating to the latest driver versions

You can find the latest selenium drivers here.

You need to update the driver versions in:

vendor/edmondscommerce/selenium-server/bin/selenium-run.bash

Running within a container

Tip

Important! Make sure to ssh-copy-id both user and root user ssh keys to container...

The easiest way to do this is to run lxc-selenium my-container which will do everything automatically.

If you wish to do this manually follow the instructions below

Configure the local machine

You will need a version of the selenium project setup and running.

This should be setup as a standalone project within the /opt/Projects folder.

Run the following

cd /opt/Projects || echo "You need to create the /opt/Projects folder";
mkdir selenium-server
cd selenium-server/
composer require edmondscommerce/selenium-server
./vendor/bin/selenium-run.bash

This will install the selenium package and try and run it. This will check to see if there are any issues around the installation, if there are fix these before continuing.

Once everything here is passing and the process starts you need to kill the process and open the remote tunnel. Run the following

lxc-remotePort my-container 4444

This will also carry out some basic checks and then open the remote connection. This will keep focus, so you need to open a new terminal and start the selenium server in that.

You should now be able to run the behat tests from within the container without requiring any further configuration changes

Running using Firefox

Warning

For the latest Geckodriver to work you need Selenium 3.3 and above.

# This will automatically download the selenium and gecko drivers
vendor/bin/selenium-background-run.bash firefox
# Run the tests using the firefox profile
vendor/bin/behat -p firefox features/

Running using Chrome Headless

To run your acceptance tests within Chrome headless, you need to add a new session to your config. If you already have a Chrome based session, you can duplicate it and add the --headless Chrome option. Otherwise, below is an example you can use.

Paste the following in to your session list.

selenium_chrome_session_headless:
               selenium2:
                browser: chrome
                capabilities:
                  extra_capabilities: { "chromeOptions": { "args": ["--headless", "--ignore-certificate-errors", "--test-type"], perfLoggingPrefs: { 'traceCategories': 'blink.console,disabled-by-default-devtools.timeline' } }, "loggingPrefs": { "performance": "ALL" } }

To use the headless mode, update your default session to use your newly defined headless session name.

Headless inside the container

If you are running Headless mode inside the container with Selenium, you will need to add the disable-gpu flag or screenshots will not work.

capabilities:
                  extra_capabilities: { "chromeOptions": { "args": ["--disable-gpu", "--headless", "--ignore-certificate-errors", "--test-type"], perfLoggingPrefs: { 'traceCategories': 'blink.console,disabled-by-default-devtools.timeline' } }, "loggingPrefs": { "performance": "ALL" } }
If you are using a fresh behat framework this should already be in place.

You will also need to install Selenium server (like on the desktop) along with Google Chrome (using the ChromeHeadless container asset). From here, you should be able to run your tests in complete isolation from your desktop machine meaning you can clone the container to run groups of tests in parallel.

Creating Context Files

This can be done by simply copying another file but you will just need to make sure you change the names of the file as this will create conflict if there are duplicates.

Here is a step-by-step guide on how to create a new file:

  1. Pick the context file you would like to copy and then place into the folder with the rest of the context folders

  2. Rename the file itself

  3. Once you are inside the new file, you will need to change the class name to the exact same as what you have given the file.

  4. Delete everything with inside the opening and closing {}

  5. Finally and most importantly be sure to add the new context file to your behat.yaml file.

Note

If the final step is not done any steps or context you write will not work.

Using Behat with Containers

Note

When working with containers, Behat may be required to develop efficiently, for either of these tools to work correctly you need to set up your interpreter configuration for the project at hand.

Edmonds Commerce Behat Framework

Edmonds Commerce has a behat framwork at https://github.com/edmondscommerce/behat-framework

Contributions can be made to this by cloning the repo and using the normal git committing strategy to make the changes available

Behat for CI

Setup

  1. Clone the container from the Cluster
    • Update the database base urls as required
    • Install any Composer dependencies
  2. Add the Chrome container asset
    • /Confidential-Categories/Confidential-Infrastructure/Clusters-and-Containers/Container-Assets/#chromeheadless
  3. Install Selenium Server inside the container

Configuration

Ensure that your behat.yaml file has a headless session and profile.

Add the block below to your behat.yaml section for sessions

selenium_chrome_session_headless:
               selenium2:
                browser: chrome
                capabilities:
                  extra_capabilities: { "chromeOptions": { "args": ["--disable-gpu", "--window-size=1920,1080", "--headless", "--ignore-certificate-errors", "--disable-background-timer-throttling", "--test-type"], perfLoggingPrefs: { 'traceCategories': 'blink.console,disabled-by-default-devtools.timeline' } }, "loggingPrefs": { "performance": "ALL" } }

And add the profile to allow us to use it, please this next to your other profiles

headless:
  extensions:
    Behat\MinkExtension:
      default_session: selenium_chrome_session_headless

This will allow us to use the profile from the commandline using -p headless which will allow us to toggle the profile in use.

Running the tests

Once setup is complete, we can run the tests as we would normally. Ensure Selenium server is running inside the container and then run your behat tests as normal.

You may need to increase your PHP memory limit, this has been added to the command for convenience.

#Ensure Selenium Server is running
bash ./bin/selenium-background-run.bash;

#Run the test suite
php -d memory_limit=2048M ./bin/behat --config ./behat.yaml -p headless -f pretty -o std

You may need to increase the memory limit for the PHP Process running Behat.

Gotchas

Behat can not access the site internally

As we are still making web requests we must ensure that the hosts file inside the container is set up correctly as we would when working outside the container.

sudo echo "127.0.0.1 mybaseurl" >> /etc/hosts; 

Selenium when not using headless mode

Now that chrome headless is the default profile you will want to run selenium from inside the container but if you would like to run selenium with visual aid you will need to do the following

Run the tests with -p chrome This will change the profile selenium uses but before you do this make sure to stop selenium from running inside the container and then run it from outside the container

  • Headless Mode = Run selenium inside the container
  • Chrome Visual Aid = Run selenium outside the container

To stop selenium use:

bash bin/selenium-stop.bash 

Strict keywords

When behat is complaining about

Step '/^I press visible button "([^"]*)"$/' was matched but the wrong keyword 'When' was used. The step should be invoked with 'Given'
Add to behat.yml
    Chekote\BehatRetryExtension:
      strictKeywords: false