By Andrew Peterson
I thought I'd write a brief article on the subject of "web
crawlers" and "spiders." I hope you'll
find it useful. If you follow a few
simple rules, you can maximize your internet exposure and move your ranking up
in the search engine results and help keep yourself near, or at, the top.
In a nutshell, companies like Google and Yahoo use search
engine programs designed to scour the internet looking for keywords, web
addresses, and traffic flows - both in and out of sites. These programs are called spiders or web
crawlers. They are a specific type of
bot, or software agent with a specialized purpose - to gather information from
websites and index it in a HUGE database for recall.
It's a super complex task due the dynamic nature of the ever
changing World Wide Web. Think about it,
how many micro changes are made every day to the tens of millions of websites
out there? It's staggering to
comprehend.
That indexed database is then accessed when a user types a
keyword into a search engine, like Google.
For example, if someone types "Laura Benedict" (without the quotes) into
Google's search engine box, the program looks for all the instances where it
finds the words "Laura Benedict." The
reason Laura's web page appears, is because her name is associated with
numerous instances where her web page address is also present. BTW Laura, you're number one on the Google
result page! You're all over it! Kudos.
That's why you'll occasionally get a dead link in a search
result. The search isn't done live, it
couldn't be! It would literally take
weeks to accomplish. The bots typically
move through the web around once a week and they don't look at more than 20% of
the entire WWW. It's just too huge to probe. But they do get a good cross section of the
most current WWW at the time they
look at it.
So here's what you can do to increase your exposure: Whenever you blog in publicly viewed forums--like
in MySpace, you should add your webpage to the end of comments you leave for
other MySpace members. Here's a simple
example:
Hi Gilligan, loved the show, especially the episode where
the ape threw the explosives at you!
All
the best, Andrew Peterson (Andy)
www.andrewpeterson.com
I didn't have to say "please visit my website"-- besides, it
rarely works anyway. When was the last
time you looked at a website because the blogger asked you to? If the reader's interested, he/she will look
at it on their own, they don't have to be asked. Make sense?
It's not considered rude or improper or BSP (Blatant Self
Promotion) to include your website in a comment to a fellow MySpacer. It's standard practice. What it does, is give the web crawlers
another spot to find your name and your web address linked together. Remember every hit counts at moving you up in the
search engine results. So whenever you're
in a public blog, forum, or you're mentioned on someone else's web page, always
ask to have a link to your website included.
It's not an unreasonable request.
Of course there are no absolutes, so you'll have to use your best judgment
when asking. To give you an actual example: When I first joined, I noticed the side bar of
the ITW Debut Author page didn't
have my website linked, so I asked Becky Cantrell to add it, which she
graciously did. Thanks, Becky!
That's also why I included my web page at the end of the
Southern California Writers Conference blog entry I made when Director, Michael
Gregory did the write-up of my website.
It looks perfectly natural to see a web address at the end of a blog
entry. As long as you don't get greedy
and say something tacky about visiting your website, it's perfectly okay. Some blog sites may have rules or
restrictions on the website link issue, so all you have to do is ask if you can
include it at the end of your blog entry.
Most will probably say yes.
I'm in a tough situation.
There's another Andrew Peterson out there who happens to be somewhat
famous. He's a gospel singer and
composer and he's also written a book.
So I've got a distinct disadvantage going. I'm hoping that will change after my book is
launched and the spiders start finding reviews, blogs, and increased web traffic
to and from my site. Until then, I just
have to be patient. What's really weird,
is that Google has somehow "blacklisted" my site -- it doesn't even show up
after 500 entries of search results.
Yahoo has me on its first page, in seventh place, last I looked (which
is actually pretty good considering FIRST TO KILL hasn't even launched
yet) I contribute some of that to all
the networking I've done to date. My web
designer is looking into the Google anomaly.
For awhile, Google had me at number four! Then I just dropped off the map. Things like this happen, so search yourself
often. (I know that sounds funny)
So, bottom line? Remember to add your web page (and in some
cases, your book's title) to all your internet traffic on blog sites, forums,
and web pages within the public realm. You
want those web crawlers and spiders finding you.
I hope this makes sense to everyone, and for those of you
who already know all of this, sorry for the redundancy of knowledge. Now go forth and get bitten by those spiders!
Andrew Peterson is the author of FIRST TO KILL, a September '08
release from Leisure Books.
www.andrewpeterson.com
