Recently the marketing team at ThoughtWorks wanted an application that can search for users on linkedin with certain specific facets to help them find potential business leads.
Although the task seemed a little drab, I realized when I started writing the code that the API Support from Linkedin was pretty poor. There were some search API wrappers that I found as ruby gems. It became pretty evident that it would actually be easier to scrape the data off the pages than to use the API to good means.
I wrote a simple crawler in ruby that can do this for me recursively. With the help of libraries like Mechanize, this task is extremely simple.
Right from the beginning, we were lazy to provide a User Interface to this application. It remained a parametrized script till completion. Then came the difficulty of teaching the marketing folks to run this on their machines. Our initial inclination was to make this a desktop application that would get all the library magic to happen during installation.
Vinkesh Banka - a colleague from Bangalore suggested that we make an automated jabber chat bot that will interactively provide the results scraped by the crawler. Brilliant!
Vinkesh’s gem Marvin is a wrapper around the xmpp library. It helped us write some really simple logic around building an ‘intelligent’ bot that will crawl linkedin and respond to queries over chat– Linbot.
Linbot on github. To see what I mean, just add linkrawl at gmail.com to your contacts and say hi to him….