Need a little writing motivation? Why not sign up for NJFM's Motivation for Writers Newsletter. You'll receive a dose of motivation delivered to your inbox every Monday.

Robots.txt    

RobotsOver the past few days I noticed my Google earnings had dropped significantly. I couldn’t figure out why the drop, but I know the Internet is a fickle beast so I didn’t lose much sleep over it.

Knowing that I can’t control Google, I continued on doing what I do, playing around behind the scenes and writing occasionally.

An Enlightening Email from Google

Imagine my surprise when I received an email from Google notifying me of “Failed Ad Crawls.” In essence their AdSense ad crawler was having a hard time crawling my sites because of my robots.txt file.

If the AdSense crawler is unable to crawl my site, Google is unable to display the most relevant ads to compliment my site’s content. This is most probably what lead to my decrease in earnings (although things were just fine a week or so ago so I’m not sure why the sudden change).

Amending the Robots.txt File

Way back when I first started playing online I was aware of a file called robots.txt file. I think I might have played around with it back then, but after switching to WordPress, I forgot all about it. Apparently WordPress installs a default robot.txt file and that’s what I had on each of my blogs.

My robots.txt file for most of my sites looked like this:

User-agent: *
 Disallow: /cgi-bin/
 Disallow: /wp-admin/

I have a few sites where the robots.txt was even more sparse and looked like this:

User-agent: *
 Disallow:

The sparse robots.txt file belongs to non-money making sites so I’m not so concerned.

Anyway, Google recommended that I add the following code to the top of my robots.txt file to allow the AdSense crawler in:Lost in Cyber Space

User-agent: Mediapartners-Google
 Disallow:

After doing a little research, I went a little further and modified my robots.txt file to look like this:

User-agent: Mediapartners-Google
 Disallow:
# Google Image
 User-agent: Googlebot-Image
 Disallow:
# global
 User-agent: *
 Disallow: /cgi-bin/
 Disallow: /wp-admin/
 Disallow: /wp-includes/
 Disallow: /wp-content/plugins/
 Disallow: /wp-content/cache/
 Disallow: /wp-content/themes/
 Disallow: /trackback/
 Disallow: /feed/
 Disallow: /comments/
Sitemap: http://www.insertsitemapurlhere.com/sitemap.xml

I don’t totally understand all of the robots.txt file stuff, but it gives me something else to research. In the meanwhile, I’ve made the changes and have noticed an increase in earnings.

A Little More Robots.txt Info

The robots.txt file is usually located in a domain’s root directory. So, if you type in the domain name followed by /robots.txt you can read anyone’s robots.txt file. Mine is here, eHow’s is here, Suite 101’s is here and CNN’s is here. There’s no secret, but you can see how each of them differ.

TweakerOver the next few days I’ll play around with my robots.txt file to see which brings in the best return. I’ll be reviewing the robots.txt files of successful sites to see how theirs read.

Golf and Online Writing

My husband always says that golf is a game of inches. I feel online writing is the same. It’s not actually inches, but a game of tweaking. If tweaking my robots.txt file will make a difference to my financial bottom line, call me a tweaker. ;)


Related Posts Plugin for WordPress, Blogger...
Bookmark and Share

Filed Under: BlogToolsWordPress

About the Author: Felicia A. Williams is a freelance writer and blogger. She spends the majority of her time with her family and writing. If she's not writing or commenting on NJFM, she's either outside smelling the roses or writing articles for one of her other sites which include Tidbits and Stuff, BLULOW, A Dose of Health and a few other sites/blogs scattered around the internet.

RSSComments (11)

Leave a Reply | Trackback URL

  1. Ken says:

    My Adsense was just disabled for “invalid activity”. I usually wouldn’t do it on someone else’s blog but please read the post below. I just can’t talk about it anymore…stupid Google.
    Ken recently posted..Hey, Google Adsense..Eff You!

  2. Samantha says:

    Thanks for the helpful post! I don’t know what surprises me more…that as writers online we’re blending more and more into programming or that Google doesn’t like our robots.
    Samantha recently posted..Is Xomba in a death spiral?

  3. Joni says:

    huh? LOL. When I was trying to finish my BS in 1991 I wish I had taken more classes in DOS (before windows). I feel as tho I would know more about the foundations of computer speak and what goes on behind the scenes. They made it so easy with the “purty pictures” in the windows that you don’t have to learn to start from :c/ (or something like that ….LOL. I still think it would help now when I read posts like this one.
    Keep on keeping on Felicia…I’ll catch on one day.
    Joni

    • Felicia says:

      LOL! I know just how you feel, Joni.

      I tend to let things go over my head until they come back to bite me in the butt. The Robots.txt bit me in the butt so I figured it was time for me to learn a little about it. :)

  4. Very useful post. I haven’t really thought about robots.txt files since I’ve been using WordPress — although I remember them from a previous web life.

    I haven’t had any problems, but I want to make sure I don’t.

    Gip
    Gip @ So Much More Life recently posted..Soul-Shaking Questions And Knowing What You Want

  5. Caleb says:

    I found out from Hostgator that you can also insert lines to tell robots how frequently they should visit your site. I had to do this as part of the steps to keep Hostgator from suspending me from going over cpu…so much for so-called unlimited hosting :lol:
    Caleb recently posted..How To Make Money From Passions

  6. Crystal says:

    I learn so much here, Felicia! I often don’t know what to do with the info, but…

    Seriously, robots.txt – who knew? I see I have some changes to make. Congrats on your earnings rebound. :)
    Crystal recently posted..Crystal Clear Edits Logo – I’m Branded Now!

  7. I have been fortunate in that I haven’t had any problems with the robot.txt files. I wouldn’t have a clue what to do if I did. It is nice to know that if there is a problem, I will probably get notified. Then I can figure out what to do like you did.

    Yes, golf is a game of inches. It is too bad that a score can be drastically affected by such a small amount, but that’s the way it is. Can’t wait for golfing weather to return.
    cashflowmantra recently posted..Make a Grand Per Month as a Stay-At-Home Parent