Here is a little bash script we knocked together to track down some malicious activity on a clients server.

Using a bit of awk etc to parse the log files we could quickly track down an IP address that was overloading the server and then take steps to block that person.

Here is the script:

#!/bin/bash

###### SETUP ############
LOG_FOLDER=/var/www/vhosts/domain.co.uk/statistics/logs
ACCESS_LOG=$LOG_FOLDER/access_log

HOW_MANY_ROWS=20000



######### FUNCTIONS ##############


function title() {
    echo "
---------------------------------
$@
---------------------------------
"
}

function urls_by_ip() {
    local IP=$1
    tail -5000 $ACCESS_LOG | awk -v ip=$IP ' $1 ~ ip {freq[$7]++} END {for (x in freq) {print freq[x], x}}' | sort -rn | head -20
}


function ip_addresses_by_user_agent(){
    local USERAGENT_STRING="$1"
    local TOP_20_IPS=&quot;<code>tail  -$HOW_MANY_ROWS $ACCESS_LOG | grep &quot;${USERAGENT_STRING}&quot;  | awk '{freq[$1]++} END {for (x in freq) {print freq[x], x}}' | sort -rn | head -20</code>&quot;
    echo &quot;$TOP_20_IPS&quot;
}

####### RUN REPORTS #############


title &quot;top 20 URLs&quot;
TOP_20_URLS=&quot;<code>tail -$HOW_MANY_ROWS $ACCESS_LOG | awk '{freq[$7]++} END {for (x in freq) {print freq[x], x}}' | sort -rn | head -20</code>&quot;
echo &quot;$TOP_20_URLS&quot;


title &quot;top 20 URLS excluding POST data&quot;
TOP_20_URLS_WITHOUT_POST=&quot;<code>tail  -$HOW_MANY_ROWS $ACCESS_LOG | awk -F&quot;[ ?]&quot; '{freq[$7]++} END {for (x in freq) {print freq[x], x}}' | sort -rn | head -20</code>&quot;
echo &quot;$TOP_20_URLS_WITHOUT_POST&quot;


title &quot;top 20 IPs&quot;
TOP_20_IPS=&quot;<code>tail  -$HOW_MANY_ROWS $ACCESS_LOG | awk '{freq[$1]++} END {for (x in freq) {print freq[x], x}}' | sort -rn | head -20</code>&quot;
echo &quot;$TOP_20_IPS&quot;

title &quot;top 20 user agents&quot;
TOP_20_USER_AGENTS=&quot;<code>tail  -$HOW_MANY_ROWS $ACCESS_LOG | cut -d\  -f12- | sort | uniq -c | sort -rn | head -20</code>&quot;
echo &quot;$TOP_20_USER_AGENTS&quot;


title &quot;IP Addresses for Top 3 User Agents&quot;

for ((I=1; I&lt;=3; I++))
do
    UA=&quot;<code>echo &quot;$TOP_20_USER_AGENTS&quot; | head -n $I | tail -n 1 | awk '{$1=&quot;&quot;; print $0}'</code>&quot;
    echo &quot;$UA&quot;
    echo &quot;~~~~~~~~~~~~~~~~~~&quot;
    ip_addresses_by_user_agent &quot;$UA&quot;
    echo &quot;
    &quot;
done

I have been working with a server that had been configured differently from the way that I prefer.

The two biggest complaints that I had about it were that the timeout was set very low, meaning that the connection would break off every five minutes, and that vim had been set up so if you selected text using the mouse you could not copy it.

The solutions to these two problems are as follows.

You can tell the terminal not to timeout with the following command

export TMOUT=0

Be aware that this will only work for the terminal that you are working with.

The issue with vim is that the mouse was triggering visual mode.

To get round this hold down shift when selecting text and everything will work as expected

If you have a bash script that you want to make sure there is only ever one instance of, for example something triggered by cron that might not have finished the next time cron tries to trigger it then you might like this little snippet:

This is built for running Magento shell scripts (if you don’t know about these, check them out) that are run on cron.

Also note the logging that keeps log files

Note the use of a character class in grep means it wont match itself – nice eh

#!/bin/bash
HOUR=<code>date +'%H:%M'</code>
RUNNING=<code>ps waux | grep &quot;longrunner[.]php&quot;</code>
if [ &quot;&quot; == &quot;$RUNNING&quot; ]
then
    echo &quot;Its not running, we can now run it&quot;
    php -f /home/my/public_html/shell/longrunner.php -- import &gt; /home/my/public_html/var/log/mylog.txt 2&gt;&amp;1
    cp -f  /home/my/public_html/var/log/my.txt /home/my/public_html/var/log/${HOUR}.my.txt
    echo &quot;COMPLETED&quot;
else
    echo &quot;It is running, aborting running this time&quot;
fi

Sometime a backup script can go wrong, and rather the overwrite the old files you place a copy of the new ones into the same folder.

This can then escalate and before you know it you have multiple levels of the same files.

If you just want to flatten these files then this script can do that you

// The duplicated directory
DUPLICATED_DIR_NAME='uploads'
// A new directory for the files to go into
NEW_DIR_NAME='realuploads'
for f in <code>find ./ | grep $DUPLICATED_DIR_NAME/$DUPLICATED_DIR_NAME </code>; 
do 
NEWFILE=<code>echo $f | sed 's/$DUPLICATED_DIR_NAME\//\//g'</code>; 
NEWDIR=../$NEW_DIR_NAME/<code>dirname $NEWFILE</code>; 
if [[ ! -d $NEWDIR ]]
then
mkdir $NEWDIR; 
fi
cp -f &quot;$f&quot; ../$NEW_DIR_NAME/$NEWFILE; 
done

If you have a load of terminals open you might find it handy to be able to rename the window title on the fly.

You can do this easily by copying this code into your ~/.bashrc file (or even pasting it into your terminal if you like)

function nameTerminal() {
    [ "${TERM:0:5}" = "xterm" ]   && local ansiNrTab=0
    [ "$TERM"       = "rxvt" ]    && local ansiNrTab=61
    [ "$TERM"       = "konsole" ] && local ansiNrTab=30 ansiNrWindow=0
        # Change tab title
    [ $ansiNrTab ] && echo -n $'\e'"]$ansiNrTab;$1"$'\a'
        # If terminal support separate window title, change window title as well
    [ $ansiNrWindow -a "$2" ] && echo -n $'\e'"]$ansiNrWindow;$2"$'\a'
}

If you have pasted this into your ~/.bashrc file you need to launch a new terminal or run:

source ~/.bashrc

From this point the function is now ready to use and you can run:

nameTerminal "My Custom Terminal Window Title"

Now you can easily choose the terminal you want based upon the window title.

Of course this then opens the door to automatically changing the window title based on all kinds of events that you might like, isn’t bash scripting fun!

I recently needed to trigger a couple of bash scripts through a web browser.

Unfortunately PHP shell_exec function grinds to a halt when it is used to trigger a long running / memory intensive script when it is used with Apache.

To get round this I instead wrote the command to a file and then wanted to trigger it using cron.

However, the commands that were being issues included a redirect and disown which were not being triggered in the following script

#!/bin/bash
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
COMMAND=<code>cat ${DIR}/commandFile | tail -n 1</code>;
#Check the command
echo ${COMMAND};
# Command is /path/to/file.sh arg1 &gt; /path/to/outputFile &amp; disown
#Run the command
${TEST}

After having a play around I found that modifying the file to this will redirect the output and then disown the process

#!/bin/bash
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
COMMAND=<code>cat ${DIR}/commandFile | tail -n 1</code>;
#Check the command
echo ${COMMAND};
# Command is /path/to/file.sh arg1 &gt; /path/to/outputFile &amp; disown
#Run the command - This line has been changed
eval ${TEST}

There is often the case when you need to remove spaces from filenames – for instance when importing broken data feeds into Magento or osCommerce systems.

The following strips spaces and replaces them with nothing :-

find directoryname -name '* *' -type f | while read f; do mv "$f" "$(echo $f | sed s/" "/""/g)";  done

or alternatively replace them with underscores :-

find directoryname -name '* *' -type f | while read f; do mv "$f" "$(echo $f | sed s/" "/"_"/g)";  done

Or any other character/string combo you like, you can even be more clever moving them out into subdirectories with more sed magic but you get the idea.  The reason the read command is there is to get the spaces in found filenames which if you simply use a “for” loop, bash splits on the spaces.

Not the most elegant way to do this, and probably could do with some extra tweaks but it works for our purposes, so presented here in case it fits yours :

# xml_value path/to/file node_key
function xml_value(){
    grep "<$2>.*<.$2>" $1 | sed -e "s/<\!\[CDATA\[//" | sed -e "s/\]\]>//" | sed -e "s/^.*<$2/<$2/" | cut -f2 -d">"| cut -f1 -d"<"
}

It also strips out the CDATA tags, which we needed to pull the database details from Magento’s local.xml

To use this to get, for example, the database host, you would use the following:

DB_HOST=$(xml_value app/etc/local.xml host)

To use this

When you are monitoring a log file you may want to narrow it down, and format the results. This simple one line command will break up the output from a log file to make it easier to quickly read


tail -f /path/to/log | grep "search term" | sed 's/\(.*\)/----------\n\1\n----------/'


Useful tip – if you’ve ever been ssh’d into a remote machine and don’t want to break the connection or open a new session just to add a new tunnel, there is a way

Press [return] then ~ then C to get to an ssh prompt, then you can add tunnels as you would at the command line e.g.
-L 80:localhost:8080

Then press [return] to return to the session you were running. Cool trick.