Here is a little bash script we knocked together to track down some malicious activity on a clients server.

Using a bit of awk etc to parse the log files we could quickly track down an IP address that was overloading the server and then take steps to block that person.

Here is the script:

#!/bin/bash

###### SETUP ############
LOG_FOLDER=/var/www/vhosts/domain.co.uk/statistics/logs
ACCESS_LOG=$LOG_FOLDER/access_log

HOW_MANY_ROWS=20000



######### FUNCTIONS ##############


function title() {
    echo "
---------------------------------
$@
---------------------------------
"
}

function urls_by_ip() {
    local IP=$1
    tail -5000 $ACCESS_LOG | awk -v ip=$IP ' $1 ~ ip {freq[$7]++} END {for (x in freq) {print freq[x], x}}' | sort -rn | head -20
}


function ip_addresses_by_user_agent(){
    local USERAGENT_STRING="$1"
    local TOP_20_IPS=&quot;<code>tail  -$HOW_MANY_ROWS $ACCESS_LOG | grep &quot;${USERAGENT_STRING}&quot;  | awk '{freq[$1]++} END {for (x in freq) {print freq[x], x}}' | sort -rn | head -20</code>&quot;
    echo &quot;$TOP_20_IPS&quot;
}

####### RUN REPORTS #############


title &quot;top 20 URLs&quot;
TOP_20_URLS=&quot;<code>tail -$HOW_MANY_ROWS $ACCESS_LOG | awk '{freq[$7]++} END {for (x in freq) {print freq[x], x}}' | sort -rn | head -20</code>&quot;
echo &quot;$TOP_20_URLS&quot;


title &quot;top 20 URLS excluding POST data&quot;
TOP_20_URLS_WITHOUT_POST=&quot;<code>tail  -$HOW_MANY_ROWS $ACCESS_LOG | awk -F&quot;[ ?]&quot; '{freq[$7]++} END {for (x in freq) {print freq[x], x}}' | sort -rn | head -20</code>&quot;
echo &quot;$TOP_20_URLS_WITHOUT_POST&quot;


title &quot;top 20 IPs&quot;
TOP_20_IPS=&quot;<code>tail  -$HOW_MANY_ROWS $ACCESS_LOG | awk '{freq[$1]++} END {for (x in freq) {print freq[x], x}}' | sort -rn | head -20</code>&quot;
echo &quot;$TOP_20_IPS&quot;

title &quot;top 20 user agents&quot;
TOP_20_USER_AGENTS=&quot;<code>tail  -$HOW_MANY_ROWS $ACCESS_LOG | cut -d\  -f12- | sort | uniq -c | sort -rn | head -20</code>&quot;
echo &quot;$TOP_20_USER_AGENTS&quot;


title &quot;IP Addresses for Top 3 User Agents&quot;

for ((I=1; I&lt;=3; I++))
do
    UA=&quot;<code>echo &quot;$TOP_20_USER_AGENTS&quot; | head -n $I | tail -n 1 | awk '{$1=&quot;&quot;; print $0}'</code>&quot;
    echo &quot;$UA&quot;
    echo &quot;~~~~~~~~~~~~~~~~~~&quot;
    ip_addresses_by_user_agent &quot;$UA&quot;
    echo &quot;
    &quot;
done

If you have a bash script that you want to make sure there is only ever one instance of, for example something triggered by cron that might not have finished the next time cron tries to trigger it then you might like this little snippet:

This is built for running Magento shell scripts (if you don’t know about these, check them out) that are run on cron.

Also note the logging that keeps log files

Note the use of a character class in grep means it wont match itself – nice eh

#!/bin/bash
HOUR=<code>date +'%H:%M'</code>
RUNNING=<code>ps waux | grep &quot;longrunner[.]php&quot;</code>
if [ &quot;&quot; == &quot;$RUNNING&quot; ]
then
    echo &quot;Its not running, we can now run it&quot;
    php -f /home/my/public_html/shell/longrunner.php -- import &gt; /home/my/public_html/var/log/mylog.txt 2&gt;&amp;1
    cp -f  /home/my/public_html/var/log/my.txt /home/my/public_html/var/log/${HOUR}.my.txt
    echo &quot;COMPLETED&quot;
else
    echo &quot;It is running, aborting running this time&quot;
fi

I recently needed to trigger a couple of bash scripts through a web browser.

Unfortunately PHP shell_exec function grinds to a halt when it is used to trigger a long running / memory intensive script when it is used with Apache.

To get round this I instead wrote the command to a file and then wanted to trigger it using cron.

However, the commands that were being issues included a redirect and disown which were not being triggered in the following script

#!/bin/bash
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
COMMAND=<code>cat ${DIR}/commandFile | tail -n 1</code>;
#Check the command
echo ${COMMAND};
# Command is /path/to/file.sh arg1 &gt; /path/to/outputFile &amp; disown
#Run the command
${TEST}

After having a play around I found that modifying the file to this will redirect the output and then disown the process

#!/bin/bash
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
COMMAND=<code>cat ${DIR}/commandFile | tail -n 1</code>;
#Check the command
echo ${COMMAND};
# Command is /path/to/file.sh arg1 &gt; /path/to/outputFile &amp; disown
#Run the command - This line has been changed
eval ${TEST}

If you have a long running script that is designed to run as cron but may be run manually from the terminal then you may want to enforce screen so that the process can’t be aborted by the SSH terminal being closed for whatever reason.

We already blogged about how to force running as screen here.

This is an extension of this that also checks for running as Cron and if so, does not force screen.

# First off, ensure running from Cron and if not, Screen
if [ -t 1 ]
then
    echo "Not running as Cron Task"
    if [ -n "$STY" ];
    then
        echo 'We are inside screen, good';
    else
        echo "Not in a screen, please run with screen"
        echo "Exiting"
        exit 1
    fi
else
    echo "Running as Cron Task"
fi

Simply paste this somewhere close to the top of your script before it does anything and this will work nicely.

If you run a Magento store or are otherwise familiar with the world of Magento then you have no doubt seen the urgent security patch that live stores needed to apply quickly.

We opted to offer the security patching service for free to our existing clients and then also to the rest of the world. The process of applying the patch is quick and painless thanks to a nice Bash script we put together so there was really no problem in offering this service. You can read all about that offer here. For anyone still not patched, the offer still stands.

Since that time we have had a few nice testimonials which we thought we would share:

Chris from Mobile CAD Surveying

Thank-you to Edmonds Commerce for highlighting a recent Magento security breach that we were unaware of. Edmonds let us know about this problem and offered to fix it for us at no cost. We gave them our log in details and within an hour a patch had been installed and everything was working fine.
Thanks once again to Edmonds for their professional service and for attending to this matter very swiftly.

Mobile CAD Surveying Ltd

Mark Noble, IT Manager at Buckley Jewellery

Buckley Jewellery originally approached Edmonds Commerce to take our existing site and develop it onto a Magento platform.
Since this first project we have used Joseph and his team for a further website and additional work to develop & expand our ecommerce business.
The offer of installing the Magento patch across our websites was typical of a company who provide an honest and refreshing approach to website design.
I would not hesitate to recommend Edmonds Commerce for any ecommerce development and advice.

Buckley Jewellery

Clare from Libertinesque

I can thoroughly recommend Joseph and his team for such a fantastic service and the security patch which was generously offered as a free service and efficiently applied with a very quick response time. Giving great personal attention every time, I have no hesitation in recommending the team for Magento upgrades and service.

Libertinesque

A lot of concern has been caused by the “Cookie Law” which says you must tell people that you use cookies if you do, and 99% of sites do!

Well here’s a very simple jQuery bolt-on that deals with that, and although visitors need javascript enabled to see it, you have made “best efforts” to inform them, and on any eCommerce site you’re going to have a “this site requires javascript message” anyway.

The file to download is from github (isn’t opensource great?) here and is as simple to implement as add the script file to the head and somewhere on the page insert the following :

<script type="text/javascript">
    $(document).ready(function() {
      $.cookieBar();
    });
</script>

The example page is also on github : http://carlwoodhouse.github.com/jquery.cookieBar/.

Wondering why your Magento admin menu is not showing up after upgrading your live Magento store?

If all attempts to get to a particular admin page with the exception of the dashboard proved abortive, this could be because there is a conflict between the Magento function that merges all Javascript files and the Apache URL rewrite. This could be fixed by turning off this Magento function, and your can run the script below to do this.

UPDATE  <code>[your_magento_database_name]</code>.<code>core_config_data</code> SET  <code>value</code> =  '0' WHERE  <code>core_config_data</code>.<code>path</code> =<code>dev/js/merge_files</code>;

We are currently offering a free service to apply the security patches for any Magento store. You can read all about that here.

For those of you who are comfortable SSHing into your server and running things manually but would like this to be as easy as possible, you can use our bash script to automatically determine which Magento security patch you require and then apply it.

Simply run the following command:

wget -qO- http://www.edmondscommerce.co.uk/stuff/applysecuritypatch.sh | /bin/sh

For information, this is the source of the above file.

#!/bin/bash

if [ ! -f app/Mage.php ]; then
  echo "Not in the magento basedir. please run from public_html, httpdocs, www or wherever index.php is."
  exit 99
fi

WGET_BINARY=$(which wget)

if [ "" = "$WGET_BINARY" ]; then
  echo "Can't find wget in path... can't continue"
  exit 98
fi

MAGENTO_VERSION=$(grep 'function getVersionInfo\(\)' -A6 app/Mage.php | sed s/[^0-9]//g | tr '\n' '.' | sed s/'\.\.*'// | sed s/'\.$'//)
echo "Your Magento version is $MAGENTO_VERSION"

export $(grep 'function getVersionInfo()' -A6 app/Mage.php | grep = | sed s/,// | sed s/\>// | sed s/'[\t ]'//g | tr "a-z" "A-Z" | sed s/^/MAGENTO_/ | sed s/"'"//g)

set | grep MAGENTO

#
#Community Edition 1.4.0.0 through 1.4.1.1 http://www.magentocommerce.com/downloads/assets/1.7.0.2/CE_1.4.0.0-1.4.1.1.patch
#Community Edition 1.4.2.0 http://www.magentocommerce.com/downloads/assets/1.7.0.2/CE_1.4.2.0.patch
#Community Edition 1.5.0.0 through 1.7.0.1 http://www.magentocommerce.com/downloads/assets/1.7.0.2/CE_1.5.0.0-1.7.0.1.patch
#

if [ "$MAGENTO_MAJOR" -eq "1" ]; then
  if [ "$MAGENTO_MINOR" -eq "4" ]; then
    if [ $MAGENTO_PATCH -lt "2" ]; then
    	CORRECT_PATCH='http://www.magentocommerce.com/downloads/assets/1.7.0.2/CE_1.4.0.0-1.4.1.1.patch'
      echo "Running Magento < 1.4.2.0 - patch is $CORRECT_PATCH";
    else
      CORRECT_PATCH='http://www.magentocommerce.com/downloads/assets/1.7.0.2/CE_1.4.2.0.patch'
      echo "Running Magento 1.4.2.0 - patch is $CORRECT_PATCH";
    fi
  elif [ $MAGENTO_MINOR -gt "4" ]; then
    if [ $MAGENTO_MINOR -lt "7" ]; then
      CORRECT_PATCH='http://www.magentocommerce.com/downloads/assets/1.7.0.2/CE_1.5.0.0-1.7.0.1.patch'
      echo "Running Magento 1.5.0.0 or above - patch is $CORRECT_PATCH";
    elif [ $MAGENTO_MINOR -eq "7" -a $MAGENTO_PATCH -eq "0" -a $MAGENTO_REVISION -lt "2" ]; then
      CORRECT_PATCH='http://www.magentocommerce.com/downloads/assets/1.7.0.2/CE_1.5.0.0-1.7.0.1.patch'
      echo "Running Magento 1.5.0.0 or above - patch is $CORRECT_PATCH";
    else
      echo "Running Magento 1.7.0.2 - already patched."
      exit 1
    fi
  fi
else 
  echo "You're not running version 1.x.x.x of Magento, I have no idea what to do!"
  exit 97
fi 

wget -O - $CORRECT_PATCH 2> /dev/null | patch -p0
if [ $? -eq 0 ]; then
  echo "Patch succeeded."
else
  echo "For some reason the patch failed.  See the output above."
  echo "You could attempt to download the patch manually and apply it - the url is :- "
  echo $CORRECT_PATCH
fi

Please note this script is supplied without any warranty, use at your own risk. We are not supplying the patch files simply making it easy to download and apply the correct one.

We strongly recommend you patch your store as soon as possible.

If you need help, just ask!

Just came across this interesting site:

https://www.onx.ms/#recipesPage

It seems to be a very nice system for really customising your Android experience.

Thinking of being able to set different Magento Categories to display its sub-categories which can be controllable in the Magento admin i.e.

from        to this;    

You might be interested in using or viewing the zipped file here;subcat