One of the most arduous task in maintaining and ecommerce web site is the constant task of keeping all of your products up to date. Suppliers like to release new price lists and product ranges on at least a yearly basis. Just when you have managed to get your web site completely up to date, the rug is pulled out from under you and you have to start all over again…

Its a never ending task and if not handled carefully can soak up ridiculous amounts of staff time. Staff time means expense as any business financial director will tell you on wages day.

There are some great gains to be made in automating the task of keeping a product catalogue up to date. The popular ecommerce platform osCommerce has a contribution called easypopulate which is an essential addition. CREloaded and other flavours of enhanced osCommerce tend to ship with easypopulate installed by default.

Its a great script, but for a very large site with a lot of demands the script does need a little extra work to make it the fully fledged backup, restore, update and insert products tool that a serious ecommerce business needs.

Most platforms either come with or have contributions or modules you can install to enable bulk importing and updating of products which definitely makes things faster. However if you are finding that you or your staff are creating these spreadsheets by hand in your spreadsheet programme then there are still amazing efficiences to be achieved.

In fact, once you get really serious, you start hitting the row limits of common spread sheet programmes which means you literally can not open the entire product feed for editing. Once you are at this kind of size (or a lot less) you will find that your computer is very sluggish and editing becomes a slow and tedious task.

A much better way is to use custom written scripts for processing your information into the correct format for your bulk updating system. This is something that Edmonds Commerce specialise in and we have many happy customers who are saving countless hours by having software for the hard work.

The ultimate solution is to recieve product information in a feed from your suppliers. This is generally a very fast and reliable way of grabbing all of the product information for a particular supplier. However not all suppliers supply a feed. Others do supply a feed, but it is missing crucial information.

In this kind of scenario, it is usually possible to actually develop a script which will grab all of the required information and images directly from your suppliers web site. The familiar copying and pasting scenario but done by a script rather than a member of staff. Thousands of products can be grabbed in an hour and all outputed into a file correctly formatted for immediate upload into your bulk updating system such as Easypopulate for oscommerce.

Again this is a real speciality of Edmonds Commerce, though you may find that the average web design company does not have this kind of service to offer as it is a bit unusual and involves a steep learning curve.

For more information on this subject please do get in touch

One of the big tasks that any ecommerce retail business must undertake is the continual updating and inserting of products into the catalogue. Done one by one, this task can take a ridiculous amount of time. In some instances there is no better option, but in the vast majority of cases there is!

Product Feed

The ideal scenario is that your supplier makes available an up to date product feed which is regularly updated and contains all of the information you need to insert those products into your catalogue. The challenge with this is that it is highly unlikely that you will literally be able to upload this data as is. The reason being that each ecommerce system has its own quirks and separate ways of doing things. Before you can upload this data into your feed, it is highly likely that it will need to be altered and prepared for insertion.

You could do updating by hand – but that brings us back to our first point. Doing things by hand can take a ridiculously large amount of time. Instead – we recommend that you have a script which does all this preparation for you.

In fact this task is something that Edmonds Commerce specialise in. Not least because it is something that we have done plenty of and so we have a good understanding of how to do the job. Furthermore – we understand how to do the job well.

Spidering and Scraping Products from Supplier Web Site

If your supplier does not provide a feed, or if the feed they supply does not have all of the information that you want, you might think you are stuck. You are not!

It is perfectly possible to build a system which will visit every product on your suppliers web site and grab all of the information and pictures and then save them into a format that you can insert into your catalogue system. It is even possible to extend the scraping system so that it goes all the way and inserts the products into your site for you.

Again this is something that Edmonds Commerce specialise in.

Conclusion

If you find that you or your staff are spending large amounts of time manually copying and pasting information from supplier web sites – you need to ask yourself if that is really cost effective. Whilst developing a script to process a feed or scrape a supplier web site might involve a significant initial outlay – the humongous saving in staff time ensures that you will quickly recoup this cost and then will be straight into a profitable scenario. Furthermore your catalogue will be absolutely up to date with the latest pictures, information and prices meaning that you have the best chance to sell those products.

If you want to discuss how Edmonds Commerce could help you achieve these great goals of cost reduction and totally up to date catalogue – please do get in touch.

One of the most useful and powerful things you can do with PHP is to create a programme which will simulate a web browser and can grab data, post data to forms and generally interact with other web sites – automatically.

For PHP to be able to work like this it must have the CURL library installed and active. It is the CURL library which actually handles all of the interaction and PHP is my scripting language of choice for interacting with CURL.

A simple CURL function is like this:

function curl($url){

$timeout = '300'; //how long before CURL gives up on this page
$go = curl_init();
curl_setopt ($go, CURLOPT_URL, $url);
curl_setopt ($go, CURLOPT_RETURNTRANSFER, 1);
curl_setopt ($go, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt ($go, CURLOPT_TIMEOUT, $timeout);
$page = curl_exec($go);
curl_close($go);
return $page;

}

This function when called and echoed will output the entire html of the $url specified.

For grabbing data from this page to be inserted into a database (for example when spidering a suppliers web site for product information to be inserted into your site) we then use regular expressions to find what we are looking for and then insert that into the database.

so for example if we wanted to grab the product title and we knew that this was wrapped in a h1 tag with the class “product title” we could use this regexp to grab this:

$page = curl($url);

$pattern = '%
<h1 class="product_title">(.+?)</h1>
%i';

preg_match($pattern,$page,$matches);

print_r($matches); //we can see the entire array of matches and choose which we want to insert into the database

We can also Post data to web sites using curl. This allows us to do all kinds of things including grabbing data that is displayed on the submission of post forms. Here is an example Curl Post Function:

function curl_post($url,$post_data){

$timeout = '300'; //how long before CURL gives up on this page
$go = curl_init();
curl_setopt ($go, CURLOPT_URL, $url);
curl_setopt ($go, CURLOPT_RETURNTRANSFER, 1);
curl_setopt ($go, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt ($go, CURLOPT_TIMEOUT, $timeout);
//now for the post section
curl_setopt($go, CURLOPT_POST, true);

curl_setopt($go, CURLOPT_POSTFIELDS, $post_data);
$page = curl_exec($go);
curl_close($go);
return $page;
}

It can be tricky to figure out exactly what data should be in the post string. To help you out though is this incredibly handy addon for firefox: Live Http Headers.

This addon lets you see exactly what is going on between your browser and the web site you are visiting. This can quickly and easily give you the information you need to replicate the same behaviour with your CURL script.

Edmonds Commerce specialise in working with PHP and CURL. If you have any spidering, screen scraping or other application that requires PHP to actively interact with other web sites – get in touch today to see how we can help you benefit from this incredibly powerful technique.

Related Resources

http://www.phpfour.com/blog/2008/01/20/php-http-class/

http://www.phpclasses.org/browse/package/1988.html

http://www.phpit.net/article/using-curl-php/

http://skeymedia.com/intro-to-curl-with-php/