Collect your contacts with the Blackbook Gem

Posted by Mike Fri, 01 Feb 2008 23:26:00 GMT

Dave Myron of Contentfree wrote a great Gem that he’s
named Blackbook. The Gem is hosted in the
Contentfree collection on Ruby
Forge. Blackbook imports
contacts from a number of sources and currently supports AOL Mail, GMail, Hotmail,
Yahoo! Mail, and contacts from CSV file. Blackbook exports the contacts that have
been collected as XML and VCard formatted text or as an Array of Hashes
with :name and :email keys.

The Blackbook architecture is straight forward and additional importers and
exporters can be written. See lib/blackbook/exporter/base.rb and
lib/blackbook/importer/base.rb for the interface definition of each. See
lib/blackbook/exporter/vcf.rb and lib/blackbook/importer/gmail.rb for
examples of exporter and importer implementations. Submit patches to Dave
for inclusion in the project.

Blackbook has a patch of Mechanize
that enables it to successfully scrape the AOL sign on service to log in.
AOL has some odd URL formatting during sign on that Mechanize does not
handle in a way that AOL expects. Dave paid a bounty to Marton Fabo to find a
solution that is in WWW::Mechanize#to_absolute_uri. See
lib/blackbook/importer/page_scraper.rb for the patch. I also was able to
help out and write most of the tests for Blackbook. Since the importers are
scraping web services I use Mocha and fixtures to mock the network interaction.
See the tests for examples of testing without having to go over the wire.

And now a simple example fetching contacts from AOL and exporting as XML all
in one command

require 'rubygems'
require 'blackbook'

# fetch aol contacts and format as xml
aol_xml = Blackbook.get :username => 'me@aol.com', 
                        :password => 'whatever',
                        :as => :xml

And a larger example.

require 'rubygems'
require 'blackbook'

aol = Blackbook.get :username => 'me@aol.com', :password => 'whatever'
gmail = Blackbook.get :username => 'me@gmail.com', :password => 'whatever'
hotmail = Blackbook.get :username => 'me@hotmail.com', :password => 'whatever'
yahoo = Blackbook.get :username => 'me@yahoo.com', :password => 'whatever'

#contacts is an array of hashes with :name and :email keys, any other metadata
#in the contacts are added as additional key/value pairs
contacts = []
# collect unique contacts by email address into an array
(aol + gmail + hotmail + yahoo).each do |contact|
  contacts << contact unless
    contacts.detect{|c| contact[:email].downcase == c[:email].downcase }
end

# export as xml formatted text
exporter = Blackbook::Exporter::Xml.new
xml = exporter.export(contacts)

# export to VCard formatted text
exporter = Blackbook::Exporter::Vcf.new
vcards = exporter.export(contacts)

The typical install:

sudo gem install blackbook

Posted in , ,  | Tags , ,  | 16 comments | no trackbacks

Comments

Leave a response

  1. Avatar Luke Francl said about 3 hours later:

    Interesting. I don’t like it when sites ask for my password on Yahoo/Gmail/etc to scrape my contacts, but it is convenient. This sure looks like it would make it easier.

  2. Avatar Matt Stone said 6 days later:

    Thanks for the library. I had some problems.. but managed to get this working by making a simple change to gmail.rb in
    the gem.

    Line 28 changed from:

    url =
    page.search(‘//meta’).first.attributes[‘content’].split(“URL=”).last
    rescue nil

    To:

    url =
    page.search(‘//meta’).first.attributes[‘content’].split(“url=”).last.gsub(“’”,"")
    rescue nil

    Just thought I’d let you know.

    rgds,
    - matt.

  3. Avatar monde said 6 days later:

    Thanks Matt.
    I have a testing script run by cron that fetches my contacts everyday to make sure blackbook is working correctly. The proverbial ‘it works for me’. Its odd that you had to run gsub on the ‘content’ value returned from the meta tag. I’m using the latest Hpricot 0.6 so I wonder if its a Hpricot issue and I that I need to make blackbook dependent upon Hpricot 0.6?

    

    mike@butch 10003 ~/projects/blackbook$ gem list -l hpricot
    

		
      
			
    • LOCAL GEMS *
      • 
		
    hpricot (0.6)
    
mike@butch 10004 ~/projects/blackbook$ gem list -r hpricot
    

		
      
			
    • REMOTE GEMS *
      • 
		
    hpricot (0.6, 0.5, 0.4)
    
hpricot-scrub (0.2.0)
    
hpricot_scrub (0.2.3, 0.2.2, 0.2.1, 0.2.0, 0.1.0)

  4. Avatar monde said 6 days later:

    and an irb session


    
mike@butch 10003 ~/projects/blackbook$ irb -rubygems -r hpricot
    
irb(main):001:0> meta = Hpricot(‘’)
    
=> #<Hpricot::Doc {emptyelem }>
    
irb(main):002:0> meta.search(‘//meta’).first.attributes[‘content’].split(‘URL=’).last
    
=> “http://mail.google.com/mail/?ui=html&amp;zy=l&#8221;
    
irb(main):003:0> quit

  5. Avatar purzelrakete said 13 days later:

    blackbook is indeed a great little gem. we just extended it with a bunch of new scrapers. check out the writeup here: http://playtype.net/past/2008/2/14/scraping_webde_gmx_and_freenet/

  6. Avatar dharana said 20 days later:

    Hi, great work but it doesn’t seem to work with hotmail when using non-hotmail accounts:

    Blackbook.get(:hotmail, { :login => ‘dharana@dharana.net’, :password => ’xxxxxx’} )

    “Blackbook::BadCredentialsError: That username and password was not accepted. Please check them and try again.”

  7. Avatar dharana said 20 days later:

    Ok, I had an error in my previous entry, it should read:

    Blackbook.get(:hotmail, { :username => ‘dharana@dharana.net’, :password => ’xxxxxx’} )

    And the error I get is: 500 => Net::HTTPInternalServerError

    c:/ruby/lib/ruby/gems/1.8/gems/mechanize-0.7.0/lib/www/mechanize.rb:166:in `get’
    c:/ruby/lib/ruby/gems/1.8/gems/blackbook-1.0.2/lib/blackbook/importer/hotmail.rb:78:in `scrape_contacts’
    c:/ruby/lib/ruby/gems/1.8/gems/blackbook-1.0.2/lib/blackbook/importer/page_scraper.rb:64:in `fetch_contacts!’
    c:/ruby/lib/ruby/gems/1.8/gems/blackbook-1.0.2/lib/blackbook/importer/base.rb:29:in `import’
    c:/ruby/lib/ruby/gems/1.8/gems/blackbook-1.0.2/lib/blackbook.rb:32:in `export’
    c:/ruby/lib/ruby/gems/1.8/gems/blackbook-1.0.2/lib/blackbook.rb:56:in `get’
    c:/ruby/lib/ruby/gems/1.8/gems/blackbook-1.0.2/lib/blackbook.rb:16:in `get’

  8. Avatar prabu.ks@gmail.com said 25 days later:

    Hi,

    I have one question.Is it possible to fetch the avator of a particular contact

  9. Avatar monde said 25 days later:

    we’ve not scraped avatars but you could hack the code to do so I would think, at least for GMail

  10. Avatar prabu.ks@gmail.com said 25 days later:

    Hi,

    I have one question.Is it possible to fetch the avator of a particular contact

  11. Avatar prabu.ks@gmail.com said 25 days later:

    I will tried for gmail in that gmail.rb file.

    the columns object fetches only two columns column1 for name and column2 for email.

  12. Avatar darudude@gmail.com said 27 days later:

    for some reason i can get it to work through command line but as soon as i try to work it in rails it doesn’t work… does anybody have sample view and controllers i can look at?

  13. Avatar Eduardo Bellani said 2 months later:

    Hello guys.
    Nice gem you have here. I’m left with a question though. I’ve managed to get this working in IRB fairly easy, also in my specs. But when it comes to the rails app, I’ve encontered this error(I’m running ruby 1.8.6 and Rails 2.0.2 with mechanize (0.7.5) and blackbook (1.0.4) )

    uninitialized constant WWW::Mechanize::SyncEnumerator

    /usr/lib/ruby/gems/1.8/gems/activesupport-2.0.2/lib/active_support/dependencies.rb:478:in `const_missing’
    /usr/lib/ruby/gems/1.8/gems/blackbook-1.0.4/lib/blackbook/importer/page_scraper.rb:13:in `to_absolute_uri’
    /usr/lib/ruby/gems/1.8/gems/mechanize-0.7.5/lib/www/mechanize.rb:176:in `get’
    /usr/lib/ruby/gems/1.8/gems/blackbook-1.0.4/lib/blackbook/importer/hotmail.rb:41:in `login’
    /usr/lib/ruby/gems/1.8/gems/blackbook-1.0.4/lib/blackbook/importer/hotmail.rb:64:in `prepare’
    /usr/lib/ruby/gems/1.8/gems/blackbook-1.0.4/lib/blackbook/importer/page_scraper.rb:63:in `fetch_contacts!’
    /usr/lib/ruby/gems/1.8/gems/blackbook-1.0.4/lib/blackbook/importer/base.rb:29:in `import’
    /usr/lib/ruby/gems/1.8/gems/blackbook-1.0.4/lib/blackbook.rb:32:in `export’
    /usr/lib/ruby/gems/1.8/gems/blackbook-1.0.4/lib/blackbook.rb:56:in `get’
    /usr/lib/ruby/gems/1.8/gems/blackbook-1.0.4/lib/blackbook.rb:16:in `get’

    Did somebody found something similar?

  14. Avatar samquidaibo@yahoo.com said 3 months later:

    Does not good. With Hotmail and Yahoo, some contacts does not have name, email or both. With Gmail, some contacts incode notice on email

  15. Avatar samquidaibo@yahoo.com said 3 months later:

    If I try to export contacts from hotmail /mail/GetContacts.aspx then the EOFError occure and blackbook get hang.

  16. Avatar monde said 3 months later:

    My plate is full with other items, perhaps I can help someone to start maintaining this gem. I know that there has been a trend away from these spammy type schemes where you invite everyone you know to join a new social service. See: http://blog.evanweaver.com/articles/2008/03/06/spokeo-spam/

Trackbacks

Use the following link to trackback from your own site:
http://plasti.cx/trackbacks?article_id=839

RSS feed for this post trackback uri

(leave url/email »)

   Comment Markup Help Preview comment
Web Statistics