webscraping « Nash's world

Tag: webscraping

IGMH mobile/web app open for testing/usage

by Nashath Rafeeq on Aug.30, 2012, under News, Programing

It was one of those rare quiet morning in office (touch wood) when Naail @kudanai (http://www.kudanai.com/) pinged me about a interesting problem he had. He asked me for suggestion on scraping IGMH doctors duty roster from this ancient looking aspx page. At the time he had google app engine app and He was attempting at the time to use YQL to rangle the data out of the page as the page was being served out of a non standard port on the server wich appengine did not like.

I suggested to him to port the app over to django proper and move the whole application over to my server. To this end we ported app over to django added some fancy (ie: stupid) caching so that we dont kill the poor server we were scraping the data from front end is built on jquerytouch and compass. API stubs are also available to any one who want to get some some sane looking data for their own apps. For access ping Naail or me (@NashRafeeg).

To to try it out for yourselves, visit: http://igmh.dot.my

Some Known Issues:

The display code is built on jQTouch, an thus inherits its flaws and shortcomings. For example, we rely on -webkit-overflow-scrolling: touch to handle the inertial scrolling. This will fail on most android devices and so on.

also, yes, we know about the “back” button issue.

The page currently doesn’t have a timed/manual refresh mechanism (which would ideally be present in the queue data view). You have to go back to home, and then back to the view in order to reload the data.

For those of you more technically inclined
The Backend is
* Nginx
* Django with FastCGI
* Mechanize
* Beautiful Soup
Front End is powered by Jquerytouch and Compass

For how we got started on this endeavor hit up http://www.kudanai.com/2012/08/igmh-mobileweb-app-open-for-testingusage.html