Plasticx Blog

Capable of being shaped or formed

Collect your contacts with the Blackbook Gem

Posted by Mike 02/01/2008 at 03:26PM

Dave Myron of Contentfree wrote a great Gem that he’s
named Blackbook. The Gem is hosted in the
Contentfree collection on Ruby
Forge. Blackbook imports
contacts from a number of sources and currently supports AOL Mail, GMail, Hotmail,
Yahoo! Mail, and contacts from CSV file. Blackbook exports the contacts that have
been collected as XML and VCard formatted text or as an Array of Hashes
with :name and :email keys.

The Blackbook architecture is straight forward and additional importers and
exporters can be written. See lib/blackbook/exporter/base.rb and
lib/blackbook/importer/base.rb for the interface definition of each. See
lib/blackbook/exporter/vcf.rb and lib/blackbook/importer/gmail.rb for
examples of exporter and importer implementations. Submit patches to Dave
for inclusion in the project.

Blackbook has a patch of Mechanize
that enables it to successfully scrape the AOL sign on service to log in.
AOL has some odd URL formatting during sign on that Mechanize does not
handle in a way that AOL expects. Dave paid a bounty to Marton Fabo to find a
solution that is in WWW::Mechanize#to_absolute_uri. See
lib/blackbook/importer/page_scraper.rb for the patch. I also was able to
help out and write most of the tests for Blackbook. Since the importers are
scraping web services I use Mocha and fixtures to mock the network interaction.
See the tests for examples of testing without having to go over the wire.

And now a simple example fetching contacts from AOL and exporting as XML all
in one command

require 'rubygems'
require 'blackbook'

# fetch aol contacts and format as xml
aol_xml = Blackbook.get :username => 'me@aol.com', 
                        :password => 'whatever',
                        :as => :xml

And a larger example.

require 'rubygems'
require 'blackbook'

aol = Blackbook.get :username => 'me@aol.com', :password => 'whatever'
gmail = Blackbook.get :username => 'me@gmail.com', :password => 'whatever'
hotmail = Blackbook.get :username => 'me@hotmail.com', :password => 'whatever'
yahoo = Blackbook.get :username => 'me@yahoo.com', :password => 'whatever'

#contacts is an array of hashes with :name and :email keys, any other metadata
#in the contacts are added as additional key/value pairs
contacts = []
# collect unique contacts by email address into an array
(aol + gmail + hotmail + yahoo).each do |contact|
  contacts << contact unless
    contacts.detect{|c| contact[:email].downcase == c[:email].downcase }
end

# export as xml formatted text
exporter = Blackbook::Exporter::Xml.new
xml = exporter.export(contacts)

# export to VCard formatted text
exporter = Blackbook::Exporter::Vcf.new
vcards = exporter.export(contacts)

The typical install:

sudo gem install blackbook

Posted in , , |

Trackbacks<

Use the following link to trackback from your own site:
http://plasti.cx/trackbacks?article_id=839

  1. Luke Francl
    02/01/2008 at 06:45PM

    Interesting. I don’t like it when sites ask for my password on Yahoo/Gmail/etc to scrape my contacts, but it is convenient. This sure looks like it would make it easier.

  2. Matt Stone
    02/07/2008 at 08:07PM

    Thanks for the library. I had some problems.. but managed to get this working by making a simple change to gmail.rb in
    the gem.

    Line 28 changed from:

    url =
    page.search(‘//meta’).first.attributes[‘content’].split(“URL=”).last
    rescue nil

    To:

    url =
    page.search(‘//meta’).first.attributes[‘content’].split(“url=”).last.gsub(“’”,"")
    rescue nil

    Just thought I’d let you know.

    rgds,
    - matt.

  3. monde
    02/07/2008 at 10:08PM

    Thanks Matt.
    I have a testing script run by cron that fetches my contacts everyday to make sure blackbook is working correctly. The proverbial ‘it works for me’. Its odd that you had to run gsub on the ‘content’ value returned from the meta tag. I’m using the latest Hpricot 0.6 so I wonder if its a Hpricot issue and I that I need to make blackbook dependent upon Hpricot 0.6?

    

    mike@butch 10003 ~/projects/blackbook$ gem list -l hpricot
    

		
      
			
    • LOCAL GEMS *
      • 
		
    hpricot (0.6)
    
mike@butch 10004 ~/projects/blackbook$ gem list -r hpricot
    

		
      
			
    • REMOTE GEMS *
      • 
		
    hpricot (0.6, 0.5, 0.4)
    
hpricot-scrub (0.2.0)
    
hpricot_scrub (0.2.3, 0.2.2, 0.2.1, 0.2.0, 0.1.0)

  4. monde
    02/07/2008 at 10:18PM

    and an irb session


    
mike@butch 10003 ~/projects/blackbook$ irb -rubygems -r hpricot
    
irb(main):001:0> meta = Hpricot(‘’)
    
=> #<Hpricot::Doc {emptyelem }>
    
irb(main):002:0> meta.search(‘//meta’).first.attributes[‘content’].split(‘URL=’).last
    
=> “http://mail.google.com/mail/?ui=html&amp;zy=l&#8221;
    
irb(main):003:0> quit

  5. purzelrakete
    02/14/2008 at 08:56AM

    blackbook is indeed a great little gem. we just extended it with a bunch of new scrapers. check out the writeup here: http://playtype.net/past/2008/2/14/scraping_webde_gmx_and_freenet/

  6. dharana
    02/21/2008 at 08:41PM

    Hi, great work but it doesn’t seem to work with hotmail when using non-hotmail accounts:

    Blackbook.get(:hotmail, { :login => ‘dharana@dharana.net’, :password => ’xxxxxx’} )

    “Blackbook::BadCredentialsError: That username and password was not accepted. Please check them and try again.”

  7. dharana
    02/21/2008 at 08:48PM

    Ok, I had an error in my previous entry, it should read:

    Blackbook.get(:hotmail, { :username => ‘dharana@dharana.net’, :password => ’xxxxxx’} )

    And the error I get is: 500 => Net::HTTPInternalServerError

    c:/ruby/lib/ruby/gems/1.8/gems/mechanize-0.7.0/lib/www/mechanize.rb:166:in `get’
    c:/ruby/lib/ruby/gems/1.8/gems/blackbook-1.0.2/lib/blackbook/importer/hotmail.rb:78:in `scrape_contacts’
    c:/ruby/lib/ruby/gems/1.8/gems/blackbook-1.0.2/lib/blackbook/importer/page_scraper.rb:64:in `fetch_contacts!’
    c:/ruby/lib/ruby/gems/1.8/gems/blackbook-1.0.2/lib/blackbook/importer/base.rb:29:in `import’
    c:/ruby/lib/ruby/gems/1.8/gems/blackbook-1.0.2/lib/blackbook.rb:32:in `export’
    c:/ruby/lib/ruby/gems/1.8/gems/blackbook-1.0.2/lib/blackbook.rb:56:in `get’
    c:/ruby/lib/ruby/gems/1.8/gems/blackbook-1.0.2/lib/blackbook.rb:16:in `get’

  8. prabu.ks@gmail.com
    02/26/2008 at 11:17PM

    Hi,

    I have one question.Is it possible to fetch the avator of a particular contact

  9. monde
    02/26/2008 at 11:30PM

    we’ve not scraped avatars but you could hack the code to do so I would think, at least for GMail

  10. prabu.ks@gmail.com
    02/27/2008 at 12:28AM

    Hi,

    I have one question.Is it possible to fetch the avator of a particular contact

  11. prabu.ks@gmail.com
    02/27/2008 at 12:59AM

    I will tried for gmail in that gmail.rb file.

    the columns object fetches only two columns column1 for name and column2 for email.

  12. darudude@gmail.com
    02/28/2008 at 04:52PM

    for some reason i can get it to work through command line but as soon as i try to work it in rails it doesn’t work… does anybody have sample view and controllers i can look at?

  13. Eduardo Bellani
    04/14/2008 at 04:15PM

    Hello guys.
    Nice gem you have here. I’m left with a question though. I’ve managed to get this working in IRB fairly easy, also in my specs. But when it comes to the rails app, I’ve encontered this error(I’m running ruby 1.8.6 and Rails 2.0.2 with mechanize (0.7.5) and blackbook (1.0.4) )

    uninitialized constant WWW::Mechanize::SyncEnumerator

    /usr/lib/ruby/gems/1.8/gems/activesupport-2.0.2/lib/active_support/dependencies.rb:478:in `const_missing’
    /usr/lib/ruby/gems/1.8/gems/blackbook-1.0.4/lib/blackbook/importer/page_scraper.rb:13:in `to_absolute_uri’
    /usr/lib/ruby/gems/1.8/gems/mechanize-0.7.5/lib/www/mechanize.rb:176:in `get’
    /usr/lib/ruby/gems/1.8/gems/blackbook-1.0.4/lib/blackbook/importer/hotmail.rb:41:in `login’
    /usr/lib/ruby/gems/1.8/gems/blackbook-1.0.4/lib/blackbook/importer/hotmail.rb:64:in `prepare’
    /usr/lib/ruby/gems/1.8/gems/blackbook-1.0.4/lib/blackbook/importer/page_scraper.rb:63:in `fetch_contacts!’
    /usr/lib/ruby/gems/1.8/gems/blackbook-1.0.4/lib/blackbook/importer/base.rb:29:in `import’
    /usr/lib/ruby/gems/1.8/gems/blackbook-1.0.4/lib/blackbook.rb:32:in `export’
    /usr/lib/ruby/gems/1.8/gems/blackbook-1.0.4/lib/blackbook.rb:56:in `get’
    /usr/lib/ruby/gems/1.8/gems/blackbook-1.0.4/lib/blackbook.rb:16:in `get’

    Did somebody found something similar?

  14. samquidaibo@yahoo.com
    04/20/2008 at 11:27PM

    Does not good. With Hotmail and Yahoo, some contacts does not have name, email or both. With Gmail, some contacts incode notice on email

  15. samquidaibo@yahoo.com
    04/21/2008 at 07:03PM

    If I try to export contacts from hotmail /mail/GetContacts.aspx then the EOFError occure and blackbook get hang.

  16. monde
    04/21/2008 at 09:23PM

    My plate is full with other items, perhaps I can help someone to start maintaining this gem. I know that there has been a trend away from these spammy type schemes where you invite everyone you know to join a new social service. See: http://blog.evanweaver.com/articles/2008/03/06/spokeo-spam/


Web Statistics