Scraping Google Adsense

By Tim Kofol

Published: November 12th, 2007

I am obsessed with Google Adsense. I waste a ton of time looking at my daily page impressions, click through percentage and earnings. Well I found a way to scrape that data to my desktop.

Thanks to schadenfreude for the basics of this idea.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

require 'rubygems'
require 'mechanize'

agent = WWW::Mechanize.new

# Adsense's login form is included in an iframe so we will need to use the iframe url instead of http://www.google.com/adsense
page = agent.get 'https://www.google.com/accounts/ServiceLoginBox?service=adsense&ltmpl=login&ifr=true&rm=hide&fpui=3&nui=15&alwf=true&passive=true&continue=https%3A%2F%2Fwww.google.com%2Fadsense%2Flogin-box-gaiaauth&followup=https%3A%2F%2Fwww.google.com%2Fadsense%2Flogin-box-gaiaauth&hl=en_US'

# The login form isnt named but it is the first and only form on the page,.. lets get it.
form = page.forms.first

# Fill out the form with your credentials and submit
form.Email = 'username'
form.Passwd = 'password'

page = agent.submit form

# This is the tricky part as we get bounced around with some javascript redirects.
# Simply follow the url. It is not dynamic, no session ids, etc so just hard code it, should never change.
page = agent.get 'https://www.google.com/accounts/CheckCookie?continue=https%3A%2F%2Fwww.google.com%2Fadsense%2Flogin-box-gaiaauth&followup=https%3A%2F%2Fwww.google.com%2Fadsense%2Flogin-box-gaiaauth&service=adsense&hl=en_US&chtml=LoginDoneHtml'

# We are now logged in and can go to any adsense page we with
page = agent.get 'https://www.google.com/adsense/report/overview'

Now you just have to scrape the page using hpricot to get the metrics you are interested in.

Combine that with geektool and now you have realtime Adsense updates straight to your desktop.

Note: Because of the way google handles adsense data, you often get inconsistent results. One second you may have 100 page views the next login you may have 50. Trust the higher number as I assume there is just a lag in adsense data being replicated on the database servers.

Sorry, comments are closed for this article.