I’ve been writing a lot about how great Bing is and made the switch to use Bing regularly but here I am in day 1 with using Bing full time and have run into problems. I am working on a Rails project doing some http requests and screen scraping. Basically, I’m extending the contacts gem to include AOL support. I wanted to see a quick reference to refresh my memory on the get / post syntax in Rails. I search Bing for “rails get http” and the first result is a link to rubyonrails.org. How useless. I search the entire first page and find absolutely nothing related to what I’m looking for. I’m beginning to second guess my previous whole-hearted support of Bing. Its good for general searches but it seems program related searches are crap with Bing. I do the same search in Google and the first result is what I wanted. Just a quick reference of someone using the Ruby get method. I tried a few other searches like ‘rails contacts gem’ and Bing doesn’t really give me much to work with in the results. I did have to dig down to the 8th result with Google but the Rails contacts gem project page is at least on the first page of the search results. Google wins when it comes to programming related searches in my opinion. Time for me to switch back to Google for the time being.

I’ve long been anti-Microsoft migrating first to Linux and then to Mac OSX. I’ve been using OpenOffice since the days it was first released (remember Star Office?). I used Mozilla long before Firefox every came out and even stopped supporting some older versions of IE with Onomojo. Needless to say, Microsoft would have to do something extremely well to catch my eye and even more so to actually get me to use it.

I have a confession to make. I’ve been using Microsoft’s Bing search engine more and more lately. I feel my results are more relevant and less spammy. It seems these days 5 out of the 10 search results on Google are simply screen scrapers reposting content and dumping Google Adwords all over the place. Of course Google is going to rank those higher simply in the name of profits. They of obviously will never admit to that though but think about it. If they generally rank sites with Adwords on them higher, they can make a significant more amount of money per day. Its hard to believe any corporation would ignore that fact. Certainly, a publicly traded company is at the mercy of shareholders who are only interested in the pursuit of more profits. With that being said, Google still has to maintain credibility in its search results so the same game applies but I simply refuse to accept the idea that Google doesn’t inflate rankings of sites that use Adwords onsite. Just do a few random searches and you’ll see what I mean.

I’ve been finding that Microsoft’s Bing search engine is pulling more relevant results and less junk results than Google. I don’t have any data to back that up. Its just my general perception of the results. One thing I really like is Bing’s image search. It does an Ajax call to keep the page scrolling down so you never have to click on the next page. Hover over an image and it magnifies it some with details. All in all, Microsoft is onto something good with their Bing search engine. I think its going to be a major competitor within the next year. I’ve already noticed that I go to Bing first for some types of searches.

People are trendy. They like the latest trends and naturally migrate towards trendy sites. Bing will win more and more search traffic if Google doesn’t up their game. Google long ago stopped actually innovating and instead went into operational mode since there’s no real competition out there. They’ve just been trying to position themselves to maintain that dominance without actually bringing anything new to the table. Bing is coming out swinging. Its only been a few months since it went live and its already making converts out of long time anti-Microsoft people like myself. In my opinion, that says a lot both about the technology behind Bing and the lack of innovation from Google. Take it for what its worth.

I don’t think its necessary to go into the benefits of adding a blog to help market your site. Its widely accepted as an easy way to add new keyword rich pages and help out rankings. There are some questions about whether an off-site blog or an on-site blog is better for rankings. When I’m referring to an on-site blog, I’m assuming its going to be integrated into the main site we’re promoting. An off-site blog might be with a blogging service like Blogger or something similar. The off-site blog will link out to the main site we’re marketing. The thinking is that that off-site blog will generate more rankings potential for the main site because it will be a valuable incoming link to the main site. While that may be true to some extent I still prefer on-site blogs.

An off-site blog may have ranking benefits by having externals links from another site into your main site but the off-site blog will require its own link building campaign independent of the main site so it can get ranked on its own. I’m not sure its a good use of resources to have 2 link building campaigns: one for the blog and one for the mian site. One benefit of an on-site blog would be that we can use the blog pages as potential landing pages for Adwords and other PPC marketing (sure we can do that with an off-site blog but it would require another click before they get to your main site). I think you could write your on-site blog posts in a way that would make the main site an informational resource for its theme. I think the SEO benefit would be better as a
result.

Keep in mind that Google hires teams of people to visit every site in their search index and rate them. The purpose is to improve the quality of search results and get spammy looking sites out of top rankings. I believe there are so many spammy looking blogs out there that just link out to other sites that their effect is decreasing over time as a result of this manual rating. The blogs like this are labeled as thin sites and devalued in ranking once they are reviewed. I think its better to focus on getting blog content on-site that makes your main site look more like an information resource. I would shy away from the traditional blog look and feel and try to make it look more like a rich resource of information about issues related to the topic. When it gets manually reviewed, you’ll more likely get a bigger thumbs up than you’d get from a thin off-site blog.

Of course this is all just an educated guess at best so take it all with a grain of salt.

For those of you paying attention to the latest changes Google has been making you may be wondering what’s been going on. Some have speculated that they’ve been cracking down on paid link exchanges for high page rank sites. That may be part of the change but it certainly isn’t all of it. Besides updating page ranks they’ve also modified the weight they give to external links. For a while they were giving huge weight to good internal linking structures so everyone starting having really good internal linking. Now they’ve shifted their focus again to increase the weight of external links to your site. This might not be a direct algorithm change but could be a side effect of a page rank update. I’m thinking that as the page ranks become stale the weights on external links diminishes because they are less certain of the reliability of those external links. When the do a page rank update those external links count more. A natural side effect of a page rank update. They count more because they just updated them so they’re much more valid than they will be four to five months from now prior to another pr update. So if your ranking dropped as a result of the latest changes (but PR stayed the same) then I suggest you focus on building some better external links into your site. If your ranking increased, don’t just sit pretty and smile at yourself because you’re getting more traffic. You need to start solidifying your position by creating more content and continue working on your internal linking strategy. If your ranking pretty much stayed the same then you need a magic 8 ball because I have no answers for you.

On a side note, from my analysis of current traffic and ranking on many different sites it appears as if Yahoo’s rankings have also been adjusted. This is less than a week after Google’s latest update. It seems odd to me that Yahoo rankings are adjusted along with Google’s. I’ve seen sites who’s page rank go up yet Google traffic go down while at the same time Yahoo traffic increases. Could this be some sort of link between Yahoo rankings and Google rankings? What purpose would Yahoo have in learning from Google ranks? Well, if they had a method to their madness then certainly they could combine learning from Google ranks with what they believe to be good and bad sites to improve their own algorithm. Search engines are always trying to improve and filter out spammy sites. Yahoo does a much better job than Google when it comes to this. Their index is slow but steady. Google freaks out over every minor change you make to your site. Put a page up and a day later you take it down? Google flips out and tosses you into the 404 trash bin of junk sites. Yahoo is much calmer and collective when it comes to their index. It takes longer to get into it but when you are in you don’t have to worry about Yahoo freaking out because your site has a 404 or two every now and then. Its one of the biggest things that upsets me about Google’s index. The web is dynamic not static. If I put a page up today and take it down tomorrow it doesn’t mean my site is junk or isn’t worthy of high rankings. It means I’m adaptable. Some things work. Some things don’t. That’s how you develop a good site. You find out what works. With Google index though, if you publish something you better be damn sure that you want it to be online forever or else.

Another note, the proper way to remove a page from the net without Google pissing on you is to first remove all links to the page. Wait a few weeks. Even though some people claim Google updates their index continuously it isn’t true. They cache pages and base their indexing off of those cached pages. You need to wait weeks before Google goes through and updates all the cached pages it has for your site. It may continuously update its index based on a tiny subset of its index (say your main page to get the latest blog) but if you remove a deeply linked page it will take a while for it to work its way out of the system. Next, you’ll want to submit a url removal request with Google using their webmaster tools. Finally, once you’ve waited long enough you can remove the page. This procedure is completely ridiculous by modern standards. Google has some serious catching up to do with other search engines in my opinion. Their indexing is inherently flawed and their results are littered with junk spammy sites. Oddly enough as far as search results go Yahoo has much more relevant results and has a much more reliable algorithm than Google could ever hope for. Google needs to take a lesson from Yahoo on search. They seem to have focused on everything but search since their IPO. Its time to get back to your roots Google and fix the problems with your search that we’ve known for some time. They’re lagging and leaving the door completely open for a rival to move in. Someone just needs a better algorithm and enough momentum.

I’ve reached my limit of frustration with current keyword trackers. The technology is simple enough that it baffles my mind why so many keyword rank trackers are for profit. There doesn’t seem to be a single decent instance of an open source keyword tracker out there that I could find. I want something open and that can run in Linux of course but my searches have left me empty handed.

I’ve started designing my own keyword tracker as a result. I will release it under the GPL because I like to keep it real like that. It will be a Rails application and I will host a version for people to use free of charge (with some limitations so it doesn’t kill my servers). Basically, you can extend the app by creating a Rails plugin for it for different search engines. I will just write one for Google for starters. Hopefully I can get some community support to get more search engines working for it. I’ve got the database mostly planned out and will be starting the project in the next week or two. I will make an instance of Trac to help the collaboration and issue tracking.

Basically, I’ll be creating something that will have multiple users. A user can login and enter a new site or track an existing site. Each site has a set of keywords which the app tracks over time. I want graphs of the keyword activity over time and I want the ability to import keywords and export the rank history. If anyone is interested in helping me out on this project just comment on this post to let me know and I’ll set you up with a Trac account so we can get started.

Google punishes sites heavily for 404 errors. By the time you realize your site has an error its usually too late and you’re already being punished for them. I suggest you stay proactive on your 404 errors and use a link checker. I found this extremely useful tool. Xenu’s Link Sleuth. It basically crawls your entire site for every single internal and external link. You can chose to ignore external links if you want as well and just focus on internal links. It even visits images and mailto and just about anything that has a ’src’ or ‘href’ in the html of your site. I considered it a nice toy when I first used it but when it quickly found numerous serious 404 issues on a few of my sites I upgraded the importance of this tool in my toolbox. This will keep you ahead of the curve instead of constantly playing catchup. Try it out on your site and I guarantee you’ll find 404s you had no idea existed. It shows you all the sites that have the bad links on them as well so you know where to go to correct the problem. Best of all this software is free! Its a Windows application unfortunately but any self respecting web developer has virtual machines with different operating systems on them, Windows being one of them, so that shouldn’t be much of a problem if you’re a serious developer.

Google called me out of the blue the other day asking if I wanted a job. It sounded like a good idea at first so I followed through with my updated resume and such. At some point they said they had 3 separate positions for me and that I should try to get in their core team first. I had a brief interview with the core team where they judged my qualifications based on 3 questions. I don’t remember what they were but I answered them all wrong. Well, the second one I didn’t even try and just said I don’t know because I was pissed that they were giving me a pop quiz and I got the first one wrong. No googling. After that interview it took less than 1 minute to find the answers. I never responded to the other requests from them because I don’t really want to work for a company who is so full of themselves that they honestly think that pop quizes are the best way to weed people out. Pass. Good luck though google. Now, onto the real meat of this post.

Google’s index seems like its continuously updated. That’s great for search if there were so many junk results.

• Internal linking – Google loves internal linking which is one reason why there are so many junk results in their search. From my experience, tossing in link dumps all over your site actually helps it do better. The result? Everyone link dumps and gets better rankings so you get a bunch of crappy search results.
• Sensitivity – For get search for now, lets focus on Google from a developer’s perspective. Google continuously updates their index. Great. What does that mean for a developer? It means that if you forget an apostrophe on an anchor tag you end up with dozens of 404s. No big deal if you catch the mistake early right? Wrong. What happens is the missing apostrophe bleeds the link on to whatever follows causing invalid links. If the Google god happens to see your mistake they will try adding those invalid links to their index. They won’t be valid so you will be penalized for having 404s on your site. That’s a sure fire way to see your rankings drop off the map for some ridiculous mistake that was corrected a few hours after it was made.
• Poor tools – Luckily Google provides you with a way to remove invalid links from their index but good luck using that thing. Lets say that one mistake created 50 404s across your site. You have to copy and paste each 404 url to the url removal form, one at a time. Not only that but you have to remove the domain name from the pasted version. So its copy paste edit, copy paste edit, 50 times in a row. Yay! Or you can copy paste them to a text editor and global replace the domain and then copy paste them into the removal form 1 by 1.
• Poor responsiveness – Ok so what? At least they provide a way to remove your urls from their index instead of waiting around for weeks right? Well, kinda of. Its not the same continuous updating that they do themselves. They’ll eventually listen to your request but only on their time. When they’re good and ready. I’ve had a request pending removal for over 2 weeks. That’s 2 weeks of being penalized for a missing apostrophe that was only live for less than 1 hour. Way to go Google.

This wouldn’t be complete without some suggestions. First, get a clue about hiring good developers. You’re going to eventually end up with a crap gene pool like Microsoft and Yahoo and be usurped by my new search engine. Next, don’t be so harsh on the occational 404 and at least provide a quick way to remove them. Your index is updated continuously, if you want feedback then use it. Don’t sit on my feedback for weeks. Next, give your web developer tools some love. They’re so primitive with little thought put in to usability. Also, internal linking shouldn’t count for nearly as much as you give credit. Look at the sites coming out these days. Huge link dumps that people pass right over. You’re forcing the creation of millions of junk sites on the internet. That’s not a good thing. And finally, your search obviously uses some type of machine learning and appears to be in a rut. People have your search figured out and are taking advantage of that. You need a smarter machine learning algorithm.

There are plenty of people talking about how you can make your Wordpress blog titles more SEO friendly. No one I’ve found however has mentioned what I’ve stumbled upon by accident. Pseudo-indented listings in Google search results. What I’ve started doing on my blogs is making my title like this:


<title>
<?php if ( is_single() ) { ?> &raquo; <? } ?> <?php wp_title(''); ?> <?php if ( is_single() ) { ?> &raquo; < ? } ?> < ?php bloginfo('name'); ?>
</title>


This adds a » in front of your post title. The result is that when the search results show up in Google your links look like it stands out more because of the » in the title. It almost makes it look like your site is more official and Google is giving you a little arrow in front of your link to prove it. It won’t effect your rankings as far as I know. Its more of a psychological advantage than anything else but I have no scientific data to back up that claim. Take a look at the title of this post to see an example title.

For some strange reason, I have this funny feeling that Google will be doing some updating this weekend. If you have an Adwords account, you may have noticed the message that some services may be unavailable on January 13. It’s been a while since the last PR update so we’re about due for something from Google. All of you who constantly watch your rankings, pay close attention in the next few weeks if its not this weekend. Its coming soon.

The internet is being filled with junk. Sure, there have always been a lot of useless pages and sites out there but there are new trends happening and we have only Google to blame. Google ranks sites based on how many links come into a site (a simplified explanation I admit). The result? The internet now has millions more web directories than it did. People in the quest for better rankings visit these sites or hire some SEO people to do it for them. There are even people promoting the addition to a large number of these web directories at once. Yay! Lets be honest. Who visits these random web directories? Sure someone may stumble upon one from a search engine and happen to click through it but its purpose and usefulness is extremely limited. I consider them junk sites that clog up my search results on occasion and make the quest for rankings as meaningless as the millions of web directories themselves. Lets fill the web with decent content like this site instead of useless junk like web directories shall we?

Another trend of junk sites are sites that just contain a bunch of Google Adsense links. These sites are everywhere these days and are always coming up in my search results on Google. These sites are even more useless than the web directories. They fill up the internet with ads that and screw up rankings and search results for everyone. If you’ve ever used Adsense and tried out the content networks you’ll see how pointless they are. Sure you’ll see a huge amount of impressions and maybe even a few click-throughs. If you actually take a closer look at that traffic and where its coming from you’ll see that it comes from these pages with just Adsense crap on them. Really good ad targeting there. Where do I sign up? I wonder if I had a few thousand Windows zombies out there if I could make easy money by having them all click-through my ads whenever their ips changed. Maybe I could just use Tor and save myself some time. I place the value of a content click-through at negative or at the most 0. I think they are just a waste of advertising money.

Which is why I have Google Adsense running at the top now? I’m not adverse to advertisements on legitimate sites that have content. Sites that are filled up with ads are not legitimate content in my opinion. Its rubbish. I think the content network is a good thing and should be more targeted to the site’s content but that’ll only work if Google could somehow get rid of all the junk sites.