Here is a little bash script we knocked together to track down some malicious activity on a clients server.

Using a bit of awk etc to parse the log files we could quickly track down an IP address that was overloading the server and then take steps to block that person.

Here is the script:

#!/bin/bash

###### SETUP ############
LOG_FOLDER=/var/www/vhosts/domain.co.uk/statistics/logs
ACCESS_LOG=$LOG_FOLDER/access_log

HOW_MANY_ROWS=20000



######### FUNCTIONS ##############


function title() {
    echo "
---------------------------------
$@
---------------------------------
"
}

function urls_by_ip() {
    local IP=$1
    tail -5000 $ACCESS_LOG | awk -v ip=$IP ' $1 ~ ip {freq[$7]++} END {for (x in freq) {print freq[x], x}}' | sort -rn | head -20
}


function ip_addresses_by_user_agent(){
    local USERAGENT_STRING="$1"
    local TOP_20_IPS=&quot;<code>tail  -$HOW_MANY_ROWS $ACCESS_LOG | grep &quot;${USERAGENT_STRING}&quot;  | awk '{freq[$1]++} END {for (x in freq) {print freq[x], x}}' | sort -rn | head -20</code>&quot;
    echo &quot;$TOP_20_IPS&quot;
}

####### RUN REPORTS #############


title &quot;top 20 URLs&quot;
TOP_20_URLS=&quot;<code>tail -$HOW_MANY_ROWS $ACCESS_LOG | awk '{freq[$7]++} END {for (x in freq) {print freq[x], x}}' | sort -rn | head -20</code>&quot;
echo &quot;$TOP_20_URLS&quot;


title &quot;top 20 URLS excluding POST data&quot;
TOP_20_URLS_WITHOUT_POST=&quot;<code>tail  -$HOW_MANY_ROWS $ACCESS_LOG | awk -F&quot;[ ?]&quot; '{freq[$7]++} END {for (x in freq) {print freq[x], x}}' | sort -rn | head -20</code>&quot;
echo &quot;$TOP_20_URLS_WITHOUT_POST&quot;


title &quot;top 20 IPs&quot;
TOP_20_IPS=&quot;<code>tail  -$HOW_MANY_ROWS $ACCESS_LOG | awk '{freq[$1]++} END {for (x in freq) {print freq[x], x}}' | sort -rn | head -20</code>&quot;
echo &quot;$TOP_20_IPS&quot;

title &quot;top 20 user agents&quot;
TOP_20_USER_AGENTS=&quot;<code>tail  -$HOW_MANY_ROWS $ACCESS_LOG | cut -d\  -f12- | sort | uniq -c | sort -rn | head -20</code>&quot;
echo &quot;$TOP_20_USER_AGENTS&quot;


title &quot;IP Addresses for Top 3 User Agents&quot;

for ((I=1; I&lt;=3; I++))
do
    UA=&quot;<code>echo &quot;$TOP_20_USER_AGENTS&quot; | head -n $I | tail -n 1 | awk '{$1=&quot;&quot;; print $0}'</code>&quot;
    echo &quot;$UA&quot;
    echo &quot;~~~~~~~~~~~~~~~~~~&quot;
    ip_addresses_by_user_agent &quot;$UA&quot;
    echo &quot;
    &quot;
done

I have been working with a server that had been configured differently from the way that I prefer.

The two biggest complaints that I had about it were that the timeout was set very low, meaning that the connection would break off every five minutes, and that vim had been set up so if you selected text using the mouse you could not copy it.

The solutions to these two problems are as follows.

You can tell the terminal not to timeout with the following command

export TMOUT=0

Be aware that this will only work for the terminal that you are working with.

The issue with vim is that the mouse was triggering visual mode.

To get round this hold down shift when selecting text and everything will work as expected

If you have a bash script that you want to make sure there is only ever one instance of, for example something triggered by cron that might not have finished the next time cron tries to trigger it then you might like this little snippet:

This is built for running Magento shell scripts (if you don’t know about these, check them out) that are run on cron.

Also note the logging that keeps log files

Note the use of a character class in grep means it wont match itself – nice eh

#!/bin/bash
HOUR=<code>date +'%H:%M'</code>
RUNNING=<code>ps waux | grep &quot;longrunner[.]php&quot;</code>
if [ &quot;&quot; == &quot;$RUNNING&quot; ]
then
    echo &quot;Its not running, we can now run it&quot;
    php -f /home/my/public_html/shell/longrunner.php -- import &gt; /home/my/public_html/var/log/mylog.txt 2&gt;&amp;1
    cp -f  /home/my/public_html/var/log/my.txt /home/my/public_html/var/log/${HOUR}.my.txt
    echo &quot;COMPLETED&quot;
else
    echo &quot;It is running, aborting running this time&quot;
fi

Sometime a backup script can go wrong, and rather the overwrite the old files you place a copy of the new ones into the same folder.

This can then escalate and before you know it you have multiple levels of the same files.

If you just want to flatten these files then this script can do that you

// The duplicated directory
DUPLICATED_DIR_NAME='uploads'
// A new directory for the files to go into
NEW_DIR_NAME='realuploads'
for f in <code>find ./ | grep $DUPLICATED_DIR_NAME/$DUPLICATED_DIR_NAME </code>; 
do 
NEWFILE=<code>echo $f | sed 's/$DUPLICATED_DIR_NAME\//\//g'</code>; 
NEWDIR=../$NEW_DIR_NAME/<code>dirname $NEWFILE</code>; 
if [[ ! -d $NEWDIR ]]
then
mkdir $NEWDIR; 
fi
cp -f &quot;$f&quot; ../$NEW_DIR_NAME/$NEWFILE; 
done

If you have a load of terminals open you might find it handy to be able to rename the window title on the fly.

You can do this easily by copying this code into your ~/.bashrc file (or even pasting it into your terminal if you like)

function nameTerminal() {
    [ "${TERM:0:5}" = "xterm" ]   && local ansiNrTab=0
    [ "$TERM"       = "rxvt" ]    && local ansiNrTab=61
    [ "$TERM"       = "konsole" ] && local ansiNrTab=30 ansiNrWindow=0
        # Change tab title
    [ $ansiNrTab ] && echo -n $'\e'"]$ansiNrTab;$1"$'\a'
        # If terminal support separate window title, change window title as well
    [ $ansiNrWindow -a "$2" ] && echo -n $'\e'"]$ansiNrWindow;$2"$'\a'
}

If you have pasted this into your ~/.bashrc file you need to launch a new terminal or run:

source ~/.bashrc

From this point the function is now ready to use and you can run:

nameTerminal "My Custom Terminal Window Title"

Now you can easily choose the terminal you want based upon the window title.

Of course this then opens the door to automatically changing the window title based on all kinds of events that you might like, isn’t bash scripting fun!

I recently needed to trigger a couple of bash scripts through a web browser.

Unfortunately PHP shell_exec function grinds to a halt when it is used to trigger a long running / memory intensive script when it is used with Apache.

To get round this I instead wrote the command to a file and then wanted to trigger it using cron.

However, the commands that were being issues included a redirect and disown which were not being triggered in the following script

#!/bin/bash
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
COMMAND=<code>cat ${DIR}/commandFile | tail -n 1</code>;
#Check the command
echo ${COMMAND};
# Command is /path/to/file.sh arg1 &gt; /path/to/outputFile &amp; disown
#Run the command
${TEST}

After having a play around I found that modifying the file to this will redirect the output and then disown the process

#!/bin/bash
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
COMMAND=<code>cat ${DIR}/commandFile | tail -n 1</code>;
#Check the command
echo ${COMMAND};
# Command is /path/to/file.sh arg1 &gt; /path/to/outputFile &amp; disown
#Run the command - This line has been changed
eval ${TEST}

If you just want to get the Magento version on the command line quickly try this one liner:

php -r "require 'app/Mage.php'; echo Mage::getVersion(); "

Does exactly what you need

There is often the case when you need to remove spaces from filenames – for instance when importing broken data feeds into Magento or osCommerce systems.

The following strips spaces and replaces them with nothing :-

find directoryname -name '* *' -type f | while read f; do mv "$f" "$(echo $f | sed s/" "/""/g)";  done

or alternatively replace them with underscores :-

find directoryname -name '* *' -type f | while read f; do mv "$f" "$(echo $f | sed s/" "/"_"/g)";  done

Or any other character/string combo you like, you can even be more clever moving them out into subdirectories with more sed magic but you get the idea.  The reason the read command is there is to get the spaces in found filenames which if you simply use a “for” loop, bash splits on the spaces.

What is Wget in general ?

Wget is a free utility for non-interactive download of file from the web. The user doesn’t need to login system every time Wget can download the entire web page or mirroring the entire web page. If download crashed or stop for various reason Wget will start download again from where it stopped. It is highly recommended for downloading file from web with slow network connections.

How to Use Wget ?

wget -t 10 www.google.com

If network connection fails Wget will to try to reconnect 20 times in default.With -t command we can specify how many times it need to reconnect.

wget -p --convert-links -r www.google.com -o logfile

This command will download the site -p and –convert will make sure all linked files are linked to downloaded document such as images and external links it enables complete offline viewing. Log file can be enabled with -O command to view the output message.

wget --spider --force-html www.google.com

–spider Will check the webpage is existent or not.
–force It will enforce the file type that have to be downloaded.

wget -u mozilla www.google.com

some site access allowed to certain user agents. So to access with certain user agent you can use this command.

Ftp connection download is achieved by

wget -r ftp://username:password@ftp.example.com

I recently had a Magento store that was not running cron jobs despite everything appearing to be set up correctly.

After tearing out my hair, I discovered that there was a cron job running that had not closed properly, which prevented any new cron jobs from starting.

This is because the cron.sh file that is used by magento checks the currently running processes, and if one exists quietly exits.

To check if this is the case you can run a modified line from the cron.sh file as set out below


ps auxwww | grep "/path/to/magento/cron.php" | grep -v grep | grep -v cron.sh


If this returns a line similar to the one below


root 2125 0.0 0.9 329896 40484 ? S Nov13 0:00 /usr/bin/php /path/to/magento/cron.php


You can kill the running job, which will allow cron to run normally. At this stage you should debug the cron tasks to see why the process stalled