Archive: scraping

I just hit an intriguing situation where a page that was perfectly viewable in my browser was not visible via curl.
I scratched my head and messed around testing the page in variety of online proxy services and local web browsers. I even stared messing about with telnet and manually typing headers. My conclusion was [...]

Sometimes you want your script to pause for a short period of time before repeating a loop or proceeding to the next step. This may be to reduce server load or even to simulate the natural pauses that a person would make whilst browsing a site. This kind of thing is especially true if you [...]

One of the most arduous task in maintaining and ecommerce web site is the constant task of keeping all of your products up to date. Suppliers like to release new price lists and product ranges on at least a yearly basis. Just when you have managed to get your web site completely up to date, [...]

One of the big tasks that any ecommerce retail business must undertake is the continual updating and inserting of products into the catalogue. Done one by one, this task can take a ridiculous amount of time. In some instances there is no better option, but in the vast majority of cases there is!
Product Feed
The ideal [...]