Sometimes I blog for the SEO industry, and sometimes I blog for small business and startup webmasters. This one will come in handy for both. Please be patient if you know a little about these types of searches. You might just pick up an extra trick or two…

Search Engine Hacks, Boolean Search Tactics, and a Few Sneaky Tricks for Competitive Intelligence

There have been books written about “Google Hacks,” but I refrain from using that term because some of these searches do little good on Google, and must be performed on other search engines, such as Yahoo.

Find Non-HTML File Types
Wouldn’t you like to get ahold of your competition’s marketing budget? Because I don’t feel like being taken to court, let’s use the US Government as our example here. Go to Google, and type the following:
whitehouse marketing budget filetype:xls
Where “whitehouse” would be replaced with your industry, this would return all of the MS Excel files indexed by Google containing the keywords “marketing budget”. The first result is pretty interesting, isn’t it: www.whitehouse.gov/omb/budget/fy2007/sheets/part.xls ?

Would you like to see a competitor’s contract? Try searching for:
seo contract filetype:doc
Where “seo” would be replaced with your industry, this would return all MS Word files indexed by Google containing the key word “contract”.

Use your imagination. There are plenty of file types out there, including txt and pdf and database files, all of which can return gold if you know how to dig.

Let Them Tell You What They Don’t Want You to See
(This trick won me fifty bucks at the SES Conference in Chicago last year. Thanks to Allen Dick of Vintage Tub and Bath)
Most people know what a robots.txt file is these days. If not, just do a search for “what is a robots.txt” and you will quickly learn. But most people don’t realize that this file is very handy for finding off-limit files. Afterall, if they don’t want the search engine to visit a certain page or section of the site, chances are, they don’t want you sniffing around in there either. Again, we don’t need any lawsuits, so let us use First Page Fitness as an example this time. You probably know that the robots.txt file is always located in the root directory. So just type into your brower’s address bar: www.domainname.com/robots.txt to see the file. In this case, you will be going to the following URL:
http://www.firstpagefitness.com/robots.txt
See anything intersting? So obviously we would not want our competitors to see the section of our site called “/client-contacts” so we have told the search engines not to index it. But we were not wise enough to password protect it (afterall, who would know about it if it couldn’t be found on a search engine?) and it’s easy enough to get to: http://www.firstpagefitness.com/client-contacts .

Finding Out Who Links to Your Competition
It has been our experience that Yahoo is best for this data, as Google tends to withhold information about your backlinks. So go to search.yahoo.com (I like this better than Yahoo.com because it’s clean and quick, like Google) and type:
linkdomain:competitorsdomain.com
You will notice that many of the links come from their own website. This skews the numbers and makes it more difficult to sort through. In the following search, we will get a list of your competitor’s backlinks EXCEPT those that come from within their own domain. It used to be that you could do these searches back-to-back by adding the second part, but nowadays Yahoo redirects linkdomain searches to their Yahoo Site Explorer section. Some people like this, but frankly it just annoys me. So delete the URL in the search bar and start over by typing:
linkdomain:competitorsdomain.com -site:competitorsdomain.com
That’s a handy way to find sites that might want to link to yours. After all, if they link to your competitor…

There are many more competitive intelligence techniques where these came from. All you have to do is use your imagination. More importantly, however, don’t make silly mistakes like putting your client contacts page online and advertising its existence to the world via a robots.txt file. Believe it or not, I have seen this happen numerous times in real world scenarios.

Stay Tuned for More Competitive Intelligence Techniques, including:
- How to tell if your competitor is cloaking (serving up one page to you, and something different to the search engine)
- What kind of redirect are they using?
- What kind of server platform are they on?
- What other sites do they have on the same server / IP Address?
- What terms are they bidding on (If shoemoney ever releases his secret weapon)?
- And more…