≡ Top Menu ≡Category Menu
You are here: Home » Robots.txt


RobotsOver the past few days I noticed my Google earnings had dropped significantly. I couldn’t figure out why the drop, but I know the Internet is a fickle beast so I didn’t lose much sleep over it.

Knowing that I can’t control Google, I continued on doing what I do, playing around behind the scenes and writing occasionally.

An Enlightening Email from Google

Imagine my surprise when I received an email from Google notifying me of “Failed Ad Crawls.” In essence their AdSense ad crawler was having a hard time crawling my sites because of my robots.txt file.

If the AdSense crawler is unable to crawl my site, Google is unable to display the most relevant ads to compliment my site’s content. This is most probably what lead to my decrease in earnings (although things were just fine a week or so ago so I’m not sure why the sudden change).

Amending the Robots.txt File

Way back when I first started playing online I was aware of a file called robots.txt file. I think I might have played around with it back then, but after switching to WordPress, I forgot all about it. Apparently WordPress installs a default robot.txt file and that’s what I had on each of my blogs.

My robots.txt file for most of my sites looked like this:

User-agent: *
 Disallow: /cgi-bin/
 Disallow: /wp-admin/

I have a few sites where the robots.txt was even more sparse and looked like this:

User-agent: *

The sparse robots.txt file belongs to non-money making sites so I’m not so concerned.

Anyway, Google recommended that I add the following code to the top of my robots.txt file to allow the AdSense crawler in:Lost in Cyber Space

User-agent: Mediapartners-Google

After doing a little research, I went a little further and modified my robots.txt file to look like this:

User-agent: Mediapartners-Google
# Google Image
 User-agent: Googlebot-Image
# global
 User-agent: *
 Disallow: /cgi-bin/
 Disallow: /wp-admin/
 Disallow: /wp-includes/
 Disallow: /wp-content/plugins/
 Disallow: /wp-content/cache/
 Disallow: /wp-content/themes/
 Disallow: /trackback/
 Disallow: /feed/
 Disallow: /comments/
Sitemap: http://www.insertsitemapurlhere.com/sitemap.xml

I don’t totally understand all of the robots.txt file stuff, but it gives me something else to research. In the meanwhile, I’ve made the changes and have noticed an increase in earnings.

A Little More Robots.txt Info

The robots.txt file is usually located in a domain’s root directory. So, if you type in the domain name followed by /robots.txt you can read anyone’s robots.txt file. Mine is here, eHow’s is here, Suite 101’s is here and CNN’s is here. There’s no secret, but you can see how each of them differ.

TweakerOver the next few days I’ll play around with my robots.txt file to see which brings in the best return. I’ll be reviewing the robots.txt files of successful sites to see how theirs read.

Golf and Online Writing

My husband always says that golf is a game of inches. I feel online writing is the same. It’s not actually inches, but a game of tweaking. If tweaking my robots.txt file will make a difference to my financial bottom line, call me a tweaker. 😉

{ 11 comments… add one }
  • cashflowmantra February 10, 2012, 1:53 pm

    I have been fortunate in that I haven’t had any problems with the robot.txt files. I wouldn’t have a clue what to do if I did. It is nice to know that if there is a problem, I will probably get notified. Then I can figure out what to do like you did.

    Yes, golf is a game of inches. It is too bad that a score can be drastically affected by such a small amount, but that’s the way it is. Can’t wait for golfing weather to return.

  • Crystal February 10, 2012, 7:58 pm

    I learn so much here, Felicia! I often don’t know what to do with the info, but…

    Seriously, robots.txt – who knew? I see I have some changes to make. Congrats on your earnings rebound. 🙂

  • Caleb February 11, 2012, 5:34 am

    I found out from Hostgator that you can also insert lines to tell robots how frequently they should visit your site. I had to do this as part of the steps to keep Hostgator from suspending me from going over cpu…so much for so-called unlimited hosting 😆

  • Gip @ So Much More Life February 11, 2012, 3:32 pm

    Very useful post. I haven’t really thought about robots.txt files since I’ve been using WordPress — although I remember them from a previous web life.

    I haven’t had any problems, but I want to make sure I don’t.


  • Joni February 11, 2012, 4:45 pm

    huh? LOL. When I was trying to finish my BS in 1991 I wish I had taken more classes in DOS (before windows). I feel as tho I would know more about the foundations of computer speak and what goes on behind the scenes. They made it so easy with the “purty pictures” in the windows that you don’t have to learn to start from :c/ (or something like that ….LOL. I still think it would help now when I read posts like this one.
    Keep on keeping on Felicia…I’ll catch on one day.

    • Felicia February 13, 2012, 4:55 pm

      LOL! I know just how you feel, Joni.

      I tend to let things go over my head until they come back to bite me in the butt. The Robots.txt bit me in the butt so I figured it was time for me to learn a little about it. 🙂

  • Samantha February 13, 2012, 12:00 am

    Thanks for the helpful post! I don’t know what surprises me more…that as writers online we’re blending more and more into programming or that Google doesn’t like our robots.

  • Ken February 15, 2012, 8:59 am

    My Adsense was just disabled for “invalid activity”. I usually wouldn’t do it on someone else’s blog but please read the post below. I just can’t talk about it anymore…stupid Google.

Leave a Comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.