As part of our Magento SEO service, the first thing we do is to make sure there are no issues with the crawlability and general health of the clients web site.

Whilst working on the Google Webmaster Tools crawl errors for a client I noticed one specific and intruiging problem for which I couldn’t immediately see the reason, everything looked to be set up perfectly.

Certain URLs were getting 404 responses. The URL was being parsed by mod_rewrite but everything looked fine so why was apache giving a 404?

The problem turns out to be that the URLs contain escaped slashes (eg search/KTA-mb667k2%2F2g),

The problem is that Apache actually handles the escaped slash and helpfully converts it to a real slash. That then means that it is trying to look in a sub folder that does not exist and hence the 404.

Chances are you don’t want escaped slashes to really be thought of as real directory separating slashes, especially if you are using mod_rewrite.

The answer is a simple one liner to be added to your vhost.conf or httpd.conf.

AllowEncodedSlashes On

There is a problem in Magento that when you try to move a category by dragging it in the admin, it can cause the above error, whilst causing the rest of the site to slow down.

This is caused by the system trying to re-index every product within the space of a transaction.

An easy way to get round this is to switch the indexing mode from auto to manual and then move the categories.

You will still need to re-index the site, but this can be run on a cron when the site is quite.

Sometime a backup script can go wrong, and rather the overwrite the old files you place a copy of the new ones into the same folder.

This can then escalate and before you know it you have multiple levels of the same files.

If you just want to flatten these files then this script can do that you

// The duplicated directory
DUPLICATED_DIR_NAME='uploads'
// A new directory for the files to go into
NEW_DIR_NAME='realuploads'
for f in <code>find ./ | grep $DUPLICATED_DIR_NAME/$DUPLICATED_DIR_NAME </code>; 
do 
NEWFILE=<code>echo $f | sed 's/$DUPLICATED_DIR_NAME\//\//g'</code>; 
NEWDIR=../$NEW_DIR_NAME/<code>dirname $NEWFILE</code>; 
if [[ ! -d $NEWDIR ]]
then
mkdir $NEWDIR; 
fi
cp -f &quot;$f&quot; ../$NEW_DIR_NAME/$NEWFILE; 
done

If you have a page that keeps getting longer because new items appear on it at the bottom and what’s at the bottom of the page is most important it can be a paint to have to keep scrolling done to see it.

You can make it auto scroll continuously with JQuery but that’s potentially really awkward if at any point you need to scroll up (it basically means you can’t).

The best solution is to make the page auto scroll in a sticky fashion. So when the page is already scrolled all the way to the bottom it auto scrolls, when it’s not at the bottom it does not scroll.

This can be achieved quite easily with the following code:

var num = (lastDocumentHeight - $(window).height()) - $(document).scrollTop();
if(num == 0) {
	$("html, body").animate({ scrollTop: $(document).height() }, "slow");
}
lastDocumentHeight = $(document).height();

In this example lastDocumentHeight is a global. It works by determining if the current position is the same as it was previously and this position equates to being at the bottom of the page then scroll. This needs to run after your page length has increased.

Sometimes Magento will some times return no results with the message “Your search returns no results” for terms that you would expect it to return many results for.
This can happen for a number of reasons including misconfiguration of search or attributes.

Unfortunately the way Magento speeds up search queries suffers from a race condition that can lead Magento to think it has cached a query when it has not and at the same time believes there are no products for that search because the cache for that query was dropped. This is a tough issue to solve but it can be done.

The two methods that work effectively to resolve the race condition problem (thus far) are to first make sure that the query cache isn’t empty before using the cached results. This works effectively because if the cache is empty it must have been dropped recently and has not been repopulated.
The other thing you can do to help reduce this further is to make Magento remembers the time at which it dropped. Then have the query caches compare their last updated time against the the last cache drop time and if the update time is before or the same as the drop time don’t use the cache.
If this isn’t completely effective it is also possible to make it not use the query cache if the time of update and the time of drop are with in a minuet of each other to help reduce this further.

I have recently been running some MySql scripts that wrote to a file.

These worked fine locally, but as soon as I deployed them I started to get the error above.

After much looking around I came across this solution.

When I was developing locally, I was connecting with a user that had global privileges. When I was running the code on the server I was connecting with a user that only had privileges for the database I was using.

The issue is that then FILE privilege is a Global setting, so the user did not have access to it, hence the access denied message.

Grant FILE privileges and you can connect as expected

If you are struggling using PHPStorm to find and replace code with Regex rules then this is your solution.

The problem is that when using the $ sign in your replacement string it confuses it because PHP Storm uses the $ sign to represent sub pattern replacements.

Take the following example code:

$data = array();
$form=$page->find('form.edit_product', 0);

//standard inputs
$inputs = $form->find('input[type="text"]');
foreach($inputs as $input){
    $data[$input->name]=$input->value;
}

//radio inputs
$inputs = $form->find('input[type="radio"]');
foreach($inputs as $input){
    if($input->checked){
        $data[$input->name]=$input->value;
    }
}

//checkbox inputs
$inputs = $form->find('input[type="checkbox"]');
foreach($inputs as $input){
    if($input->checked){
        $data[$input->name]=$input->value;
    }
}

//textareas
$textareas = $form->find('textarea');
foreach($textareas as $textarea){
    $data[$textarea->name]=$textarea->innertext;
}

And trying to replace the key being used in the data array with a processed one calling a method $this-dataName($key) to generate the following code:

$data = array();
$form=$page->find('form.edit_product', 0);

//standard inputs
$inputs = $form->find('input[type="text"]');
foreach($inputs as $input){
    $data[$this->dataName($input->name)]=$input->value;
}

//radio inputs
$inputs = $form->find('input[type="radio"]');
foreach($inputs as $input){
    if($input->checked){
        $data[$this->dataName($input->name)]=$input->value;
    }
}

//checkbox inputs
$inputs = $form->find('input[type="checkbox"]');
foreach($inputs as $input){
    if($input->checked){
        $data[$this->dataName($input->name)]=$input->value;
    }
}

//textareas
$textareas = $form->find('textarea');
foreach($textareas as $textarea){
    $data[$this->dataName($textarea->name)]=$textarea->innertext;
}

You might try the find pattern:

\$data\[\$([^-]+)->name\]

And the replace pattern:

\$data\[\$this->dataName(\$$1->name)\]

However this will give you the dreaded “malformed replacement string” Error

The solution is simply to triple escape your dollar signs, so the replacement pattern becomes:

\\\$data\[\\\$this->dataName(\\\$$1->name\)\]

And it works, woot!

Also when it does work, PHP Storms replacement preview feature is really quite nice

I recently needed to trigger a couple of bash scripts through a web browser.

Unfortunately PHP shell_exec function grinds to a halt when it is used to trigger a long running / memory intensive script when it is used with Apache.

To get round this I instead wrote the command to a file and then wanted to trigger it using cron.

However, the commands that were being issues included a redirect and disown which were not being triggered in the following script

#!/bin/bash
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
COMMAND=<code>cat ${DIR}/commandFile | tail -n 1</code>;
#Check the command
echo ${COMMAND};
# Command is /path/to/file.sh arg1 &gt; /path/to/outputFile &amp; disown
#Run the command
${TEST}

After having a play around I found that modifying the file to this will redirect the output and then disown the process

#!/bin/bash
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
COMMAND=<code>cat ${DIR}/commandFile | tail -n 1</code>;
#Check the command
echo ${COMMAND};
# Command is /path/to/file.sh arg1 &gt; /path/to/outputFile &amp; disown
#Run the command - This line has been changed
eval ${TEST}

If you have a Magento site that is consistently erroring out and giving you the report screen with the file name, but the file does not exist in the reports folder then this could be your solution.

The most likely culprit in this scenario is that the filesystem has run out of space. If this happens, Magento will die because it literally can’t create any files and it needs to do this for caching, session etc.

You can easily check this by running this command:

df -h

If you see any filesystems with 100% usage then you have your answer, you have run out of hard drive space and need to free some space up by deleting files.

On occasion though you might run this and see that you still have plenty of space. This can be puzzling at first though you need to bear in mind that Linux filesystems don’t purely use space but also something called inodes.

An inode is like a file pointer, you need one for every file that is in the filesystem. Also there is a limit to the number of inodes in any file system. If something goes crazy and your system creates lots of small files then you might find that you run out of inodes before you run out of hard drive space.

To check if this is your issue run this command:

df -i

If you see any lines with 100% then you have run out of inodes and you need to delete some files. There are numerous possible causes of this and tactics that can be used to clear them out. When dealing with Magento though there are a few usual suspects.

Magento is a real culprit for this behaviour as it uses file system based sessions by default. Session files are very small and can be very numerous.

The quick fix is just to completely remove the sessions directory:

rm -rf /path/to/magento/var/session

Magento will recreate it on the next request.

If you would like to try to clear out just older sessions then you can do something like this:

find /path/to/magento/var/session -name 'sess_*' -type f -mtime +7 -exec rm {} \;

OK so you love Magento’s shell scripts! They are great for quickly putting together bulk processes that have full access to the Magento ORM.

These kinds of shell scripts will often require above normal amounts of RAM for the kind of heavy lifting they lend themselves well to. If you are finding your script is banging against some mysterious memory limit that really shouldn’t be there then perhaps this is your answer:

You might see an error message such as this:

Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 20 bytes)

The reason this might be happening is that one of the things the abstract shell class does is parse the .htaccess file and apply any PHP configuration it finds. This is pretty slick and I can definitely see the value of it however if you are not aware its happening and you have a memory limit defined in a .htaccess file then it will be applied to your shell environment.

The solution is up to you, edit the htaccess file or override this method in your shell class so that it doesn’t do things you do not want:

    /**
     * Parse .htaccess file and apply php settings to shell script
     *
     */
    protected function _applyPhpVariables()
    {
        $htaccess = $this->_getRootPath() . '.htaccess';
        if (file_exists($htaccess)) {
            // parse htaccess file
            $data = file_get_contents($htaccess);
            $matches = array();
            preg_match_all('#^\s+?php_value\s+([a-z_]+)\s+(.+)$#siUm', $data, $matches, PREG_SET_ORDER);
            if ($matches) {
                foreach ($matches as $match) {
                    @ini_set($match[1], $match[2]);
                }
            }
            preg_match_all('#^\s+?php_flag\s+([a-z_]+)\s+(.+)$#siUm', $data, $matches, PREG_SET_ORDER);
            if ($matches) {
                foreach ($matches as $match) {
                    @ini_set($match[1], $match[2]);
                }
            }
        }
    }