Curl | Edmonds Commerce Blog

Archive for the 'curl' Category

PHP Save Images Using cURL

Tuesday, March 18th, 2008

Some hosts disabled the ini setting allow_url_fopen. This also means that the ability to easily grab images by calling imagecreatefromjpeg($img) where $img is a url for an external image does not work.

This is a shame as this technique is pretty easy..

PLAIN TEXT

PHP:

$remote_img = 'http://www.somwhere.com/images/image.jpg';
$img = imagecreatefromjpeg($remote_img);
$path = 'images/';
imagejpeg($img, $path);

However I was tearing my hair out trying to get a product feed integration system to work on a host that has disabled the allow_url_fopen mechanism which meant the above wouldnt work.

Thankfully cURL was still working. So here is the function I made for grabbing and saving an image using cURL instead:

PLAIN TEXT

PHP:

//Alternative Image Saving Using cURL seeing as allow_url_fopen is disabled - bummer
function save_image($img,$fullpath){
$ch = curl_init ($img);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_BINARYTRANSFER,1);
$rawdata=curl_exec($ch);
curl_close ($ch);
if(file_exists($fullpath)){
unlink($fullpath);
}
$fp = fopen($fullpath,'x');
fwrite($fp, $rawdata);
fclose($fp);
}

Another Problem Solved

Posted in curl, php | No Comments »

Using cURL with XAMPP

Wednesday, February 27th, 2008

If you have read through this blog, you will realise that I am a big fan of the cURL library for PHP. Also, when using windows to code - I like to use the XAMPP package to give me easy access to an Apache, PHP and MySQL stack.

So here is my quick guide to getting XAMPP to work with cURL. Its really easy:

1. Navigate to your root XAMPP folder

2. Search for all files and folder with this search query: php*.ini

3. Open up all of the files in your favourite windows text editor (try notepad++)

4. Do a find and replace: Find ;extension=php_curl.dll and replace with extension=php_curl.dll (you are just removing the semi colon). Make sure you select to replace in all opened documents

5. File -> Save All Documents

6. Restart Apache using the XAMPP control panel

Thats it - enjoy using cURL.

http://www.tildemark.com/programming/php/enable-curl-with-xampp-on-windows-xp.html

http://www.menyhart.net/blog/developer-stuff/enabling-curl-on-xampp/

http://incoherentbabble.com/2007/04/21/using-curl-in-xampp/

http://gabrieljones.com/getting-curl-to-work-in-xampp/

http://rovani.net/weblog/?p=19

Posted in curl, xampp | 2 Comments »

Building Spiders: Grab Data, Post Forms and Interact with Web Sites Automatically

Thursday, February 14th, 2008

One of the most useful and powerful things you can do with PHP is to create a programme which will simulate a web browser and can grab data, post data to forms and generally interact with other web sites - automatically.

For PHP to be able to work like this it must have the CURL library installed and active. It is the CURL library which actually handles all of the interaction and PHP is my scripting language of choice for interacting with CURL.

A simple CURL function is like this:

PLAIN TEXT

PHP:

function curl($url){
$timeout = '300'; //how long before CURL gives up on this page
$go = curl_init();
curl_setopt ($go, CURLOPT_URL, $url);
curl_setopt ($go, CURLOPT_RETURNTRANSFER, 1);
curl_setopt ($go, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt ($go, CURLOPT_TIMEOUT, $timeout);
$spage = curl_exec($go);
curl_close($go);
return $page;
}

This function when called and echoed will output the entire html of the $url specified.

For grabbing data from this page to be inserted into a database (for example when spidering a suppliers web site for product information to be inserted into your site) we then use regular expressions to find what we are looking for and then insert that into the database.

so for example if we wanted to grab the product title and we knew that this was wrapped in a h1 tag with the class "product title" we could use this regexp to grab this:

PLAIN TEXT

PHP:

$page = curl($url);
$pattern = '%
<h1 class="product_title">(.+?)</h1>
%i';
preg_match($pattern,$page,$matches);
print_r($matches); //we can see the entire array of matches and choose which we want to insert into the database

We can also Post data to web sites using curl. This allows us to do all kinds of things including grabbing data that is displayed on the submission of post forms. Here is an example Curl Post Function:

PLAIN TEXT

PHP:

function curl_post($url,$post_data){
$timeout = '300'; //how long before CURL gives up on this page
$go = curl_init();
curl_setopt ($go, CURLOPT_URL, $url);
curl_setopt ($go, CURLOPT_RETURNTRANSFER, 1);
curl_setopt ($go, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt ($go, CURLOPT_TIMEOUT, $timeout);
//now for the post section
curl_setopt($go, CURLOPT_POST, true);
curl_setopt($go, CURLOPT_POSTFIELDS, $post);
$spage = curl_exec($go);
curl_close($go);
return $page;
}

It can be tricky to figure out exactly what data should be in the post string. To help you out though is this incredibly handy addon for firefox: Live Http Headers.

This addon lets you see exactly what is going on between your browser and the web site you are visiting. This can quickly and easily give you the information you need to replicate the same behaviour with your CURL script.

Edmonds Commerce specialise in working with PHP and CURL. If you have any spidering, screen scraping or other application that requires PHP to actively interact with other web sites - get in touch today to see how we can help you benefit from this incredibly powerful technique.

Related Resources

http://www.phpfour.com/blog/2008/01/20/php-http-class/

http://www.phpclasses.org/browse/package/1988.html

http://www.phpit.net/article/using-curl-php/

http://skeymedia.com/intro-to-curl-with-php/

Posted in curl, firefox, php, programming, spidering | No Comments »