If you use SagePay Form integration then you might think that it is impossible to release orders created via this integration system via the API.

Historically this was true however thanks to the new admin and reporting API you can now grab all of the details you need to perform the kind of admin side integration previously only possible via Server and Direct integrations.

This does take a bit of work but if you like the fact that Form alleviates PCI compliance issues, allows your customers to use the nice easy to use Form interface and also gives you easy PayPal integration then this is a great solution.

Thanks goes out to SagePay’s technical support who have proved really helpful in ironing out the kinks in getting this working.

Here is a little bash script we knocked together to track down some malicious activity on a clients server.

Using a bit of awk etc to parse the log files we could quickly track down an IP address that was overloading the server and then take steps to block that person.

Here is the script:

#!/bin/bash

###### SETUP ############
LOG_FOLDER=/var/www/vhosts/domain.co.uk/statistics/logs
ACCESS_LOG=$LOG_FOLDER/access_log

HOW_MANY_ROWS=20000



######### FUNCTIONS ##############


function title() {
    echo "
---------------------------------
$@
---------------------------------
"
}

function urls_by_ip() {
    local IP=$1
    tail -5000 $ACCESS_LOG | awk -v ip=$IP ' $1 ~ ip {freq[$7]++} END {for (x in freq) {print freq[x], x}}' | sort -rn | head -20
}


function ip_addresses_by_user_agent(){
    local USERAGENT_STRING="$1"
    local TOP_20_IPS=&quot;<code>tail  -$HOW_MANY_ROWS $ACCESS_LOG | grep &quot;${USERAGENT_STRING}&quot;  | awk '{freq[$1]++} END {for (x in freq) {print freq[x], x}}' | sort -rn | head -20</code>&quot;
    echo &quot;$TOP_20_IPS&quot;
}

####### RUN REPORTS #############


title &quot;top 20 URLs&quot;
TOP_20_URLS=&quot;<code>tail -$HOW_MANY_ROWS $ACCESS_LOG | awk '{freq[$7]++} END {for (x in freq) {print freq[x], x}}' | sort -rn | head -20</code>&quot;
echo &quot;$TOP_20_URLS&quot;


title &quot;top 20 URLS excluding POST data&quot;
TOP_20_URLS_WITHOUT_POST=&quot;<code>tail  -$HOW_MANY_ROWS $ACCESS_LOG | awk -F&quot;[ ?]&quot; '{freq[$7]++} END {for (x in freq) {print freq[x], x}}' | sort -rn | head -20</code>&quot;
echo &quot;$TOP_20_URLS_WITHOUT_POST&quot;


title &quot;top 20 IPs&quot;
TOP_20_IPS=&quot;<code>tail  -$HOW_MANY_ROWS $ACCESS_LOG | awk '{freq[$1]++} END {for (x in freq) {print freq[x], x}}' | sort -rn | head -20</code>&quot;
echo &quot;$TOP_20_IPS&quot;

title &quot;top 20 user agents&quot;
TOP_20_USER_AGENTS=&quot;<code>tail  -$HOW_MANY_ROWS $ACCESS_LOG | cut -d\  -f12- | sort | uniq -c | sort -rn | head -20</code>&quot;
echo &quot;$TOP_20_USER_AGENTS&quot;


title &quot;IP Addresses for Top 3 User Agents&quot;

for ((I=1; I&lt;=3; I++))
do
    UA=&quot;<code>echo &quot;$TOP_20_USER_AGENTS&quot; | head -n $I | tail -n 1 | awk '{$1=&quot;&quot;; print $0}'</code>&quot;
    echo &quot;$UA&quot;
    echo &quot;~~~~~~~~~~~~~~~~~~&quot;
    ip_addresses_by_user_agent &quot;$UA&quot;
    echo &quot;
    &quot;
done

I have been working with a server that had been configured differently from the way that I prefer.

The two biggest complaints that I had about it were that the timeout was set very low, meaning that the connection would break off every five minutes, and that vim had been set up so if you selected text using the mouse you could not copy it.

The solutions to these two problems are as follows.

You can tell the terminal not to timeout with the following command

export TMOUT=0

Be aware that this will only work for the terminal that you are working with.

The issue with vim is that the mouse was triggering visual mode.

To get round this hold down shift when selecting text and everything will work as expected

What is Wget in general ?

Wget is a free utility for non-interactive download of file from the web. The user doesn’t need to login system every time Wget can download the entire web page or mirroring the entire web page. If download crashed or stop for various reason Wget will start download again from where it stopped. It is highly recommended for downloading file from web with slow network connections.

How to Use Wget ?

wget -t 10 www.google.com

If network connection fails Wget will to try to reconnect 20 times in default.With -t command we can specify how many times it need to reconnect.

wget -p --convert-links -r www.google.com -o logfile

This command will download the site -p and –convert will make sure all linked files are linked to downloaded document such as images and external links it enables complete offline viewing. Log file can be enabled with -O command to view the output message.

wget --spider --force-html www.google.com

–spider Will check the webpage is existent or not.
–force It will enforce the file type that have to be downloaded.

wget -u mozilla www.google.com

some site access allowed to certain user agents. So to access with certain user agent you can use this command.

Ftp connection download is achieved by

wget -r ftp://username:password@ftp.example.com

If you are tearing your hair out trying to figure out why you are not able to log into the Magento admin on some browsers then this might be your solution.

This solution will apply particularly if you are setting up Magento on a brand new server or VPS as you will see.

The problem is actually that the server time is wrong. The server time is used for generating cookies and the fact that the time is wrong means that cookies become invalidated immediately. Some browsers will deal with this in a less aggressive way than others and might actually let the cookie work, others will go for hard line security and will make the whole thing break.

The way to check this is to SSH into the server and check the time:

date

If the time is wrong then you need to do the following to fix it:

yum install ntp
chkconfig ntpd on
ntpdate pool.ntp.org
/etc/init.d/ntpd start

The above will install NTP, make it run on startup, will update the time and will set it running so the time should stay up to date from here on.

To check that has worked simply check the time again:

date

Then finally flush your Magento cache and sessions

rm -rf var/cache/*
rm -rf var/sessions/*

And you should be good to go

What is Rsync in general ?

Rsync works like the rcp command in linux. It is used to copy files across machines. It uses efficient checksum algorithm that looks for modified files by time and size. It is used for two purposes:

Backup

Rsync is used to transfer file from the local to a remote machine. By default Rsync will use rsh, however you can instead use ssh for high security. Compressed files are transferred to save file transfer time and reduce the load on the network. It helps to transfer bulk files in minutes.

Mirroring

Rsync can synchronize file between two servers location(i.e) local and remote. By using the checksum algorithm it updates local files only if it is updated in remote server.

How to use Rsync

rsync -avze ssh -p2020 root@server.com:/sitepath/files   /localdata

This command transfer files from remote server to local machine.

a Archive. It is denotes all related files and documents.
v Verbose.It will display status message as it is processing.
z Compression.It achives better compression ratio.
e It allows to chose alternative remote shell program for communication.

rsync -avze ssh -p2020 --exclude home/var/cache root@server.com:/sitepath/files /localdata

This command will transfer files from remote server to local machine by excluding cache folder.

rsync -avz /sitepath/files root@server.com:/localdata 

This command will transfer all file from local to remote server. It is uses default remote shell protocol.

What is an SSH access?

SSH stands for Secure Socket Shell. SSH is a protocol, like HTTP, it gives us a way to gain a secure access to remote computers.

The principle is to connect two networked computers via a secure encrypted channel within an insecure network environment such as the Internet.

With SSH access we can easily move files around and run programs on your server. We can also transmit files to and from our own network onto your server, for example to take a copy of your site for local development or to push updated files onto your server.

SSH needs a server application installing on your server (most web servers will have this by default) and an SSH client on your local machine to allow you to connect to your server.

Where do you get your SSH access from?

If you need SSH access details you should ask your hosting company, they should be able to install it for you if it is not already in place. If you are on shared hosting then the chances are that you will not be able to get SSH access details, if you need SSH access then you should consider upgrading your hosting to something that will support it.

SSH access details will consist of at least:

domain or IP address:

domain.com

123.123.123.123

username:

mywebserverlogin

password:

ihy324579ySDWf

it may also include a port if SSH is not running on the standard port.

2020

Why do we need the SSH access details?

We will always ask our customers to provide us with SSH access details. This is for two main reasons:

1. Power and Speed

SSH access allows us to run commands directly on the server, compress files, run scripts etc. For example we will generally run some static analysis scripts for our Magento customers using a bash script that we can run with SSH access. This gives us some instant feedback on the state of your Magento installation and allows us to spot any potential problems.

If we need to get a copy of your site files, using SSH we can prepare a compressed archive of your site files allowing us to download them much faster. Likewise we can easily get a reliable database dump, especially if you have a large database that might cause problems for browser based tools such as phpMyAdmin.

2. Security

SSH is a much more secure protocol than something like FTP. In SSH everything is encrypted. It has been built from the ground up to be a very secure way of networking and inter communicating between servers on an insecure network such as the web.

So sometimes you might have a restored backup without the correct permissions (whether that be because the backup didn’t store the permissions, the file system refused to take them, or you simply forgot to add the -p flag to tar!) and you want to copy the permissions from one machine to another.

Here’s a little script to do just that that we found whilst browsing the internet to do just that (uses perl and find) :-

find / -type f -o -type d 2>/dev/null | while read FILE; do   perl -le'
    printf "chmod %o \"%s\"\n", 07777 & (stat)[2], $_
    for @ARGV
  ' "$FILE"; done > /tmp/file_perms.sh

If you run that on the server with the correct permissions, then transfer the file to the server with the wrong permissions and run the file with the following command, it will fix the perms on that server :-

sh /tmp/file_perms.sh

Which may save a lot of transfer time restoring a new backup.

When you are monitoring a log file you may want to narrow it down, and format the results. This simple one line command will break up the output from a log file to make it easier to quickly read


tail -f /path/to/log | grep "search term" | sed 's/\(.*\)/----------\n\1\n----------/'


If you are scratching your head trying to figure out why you keep getting MySQL server gone away error messages despite the fact you have bumped up all the timeout etc configurations to high values then this could be your solution.

MySQL will also give you this error if you try to send a packet that is larger than the packet size defined. We had an application that was using MariaDB which has a default max allowed packet of 16M by default.

The application in question was sending large amounts of data to be stored and so the solution to the gone away issues was simply to increase the max_allowed_packet configuration in my.cnf, restart the mysql daemon and the problems are sorted.