Many sites go through redesigns and in the process URLs change. In particular, most sites we redesign we attempt to use pretty urls that contain keywords for the page. The keywords end up bold in search results if they are in the search terms. The thinking is that the bolding of the keywords increases your CTR on search results. The problem comes when you try to figure out what to do with a site that is already ranked well and you are switching over to pretty urls. The question is whether you should let the old urls just 404 or should you go through and make 301 redirects to the new pretty urls. Here are my thoughts on it:

The search engines will see the 301 redirects and start to modify their index with the new urls. The entire site won’t be reindexed at once so for a period of time it will still think that there are current pages on the site linking to urls that then do a 301. This will negatively impact rankings. How much impact is difficult to say. As the entire site is eventually reindexed the search engines will see that the site no longer links to the old urls and the site won’t be negatively impacted. I’m guessing it’ll take 3 months to remove any negative impact. Its just an educated guess though.

The 301 redirects will also have a positive impact on rankings in that the urls will contain the relevant keywords. As the site is gradually reindexed, the search engines will start to see that there is a navigational importance to the keywords used in the links on the site. It will boost the effect of those keywords and improve how the search engines interpret the site’s internal linking structure. As a result, it will have a positive impact.

Once the site is completely reindexed and the negative impact from the 301 redirects is negligible the site will ultimate have better ranking potential than before the change since the internal linking structure has been given more weight to keywords in navigational structure.

This is all known and predictable. What we can’t be sure of is if the weight of the positive impact will balance the weight of the negative impact in the short term while the site is being reindexed. From my experience, I expect rankings to slide initially and over a three month period regain and surpass the original rankings.

If you simply ignore the current site urls and just do 404s, the search engines will start to reindex your site but will also register a bunch of 404s. This is probably going to give you a much more negative impact than you’ll get from the small decline you’ll get from doing 301s. I strongly advise doing 301 redirects and to remember that your rankings will likely drop slightly but will ultimately recover and exceed where they were before (that’s assuming there wasn’t any changes in the redesign itself that negatively effects your rankings).

I don’t think its necessary to go into the benefits of adding a blog to help market your site. Its widely accepted as an easy way to add new keyword rich pages and help out rankings. There are some questions about whether an off-site blog or an on-site blog is better for rankings. When I’m referring to an on-site blog, I’m assuming its going to be integrated into the main site we’re promoting. An off-site blog might be with a blogging service like Blogger or something similar. The off-site blog will link out to the main site we’re marketing. The thinking is that that off-site blog will generate more rankings potential for the main site because it will be a valuable incoming link to the main site. While that may be true to some extent I still prefer on-site blogs.

An off-site blog may have ranking benefits by having externals links from another site into your main site but the off-site blog will require its own link building campaign independent of the main site so it can get ranked on its own. I’m not sure its a good use of resources to have 2 link building campaigns: one for the blog and one for the mian site. One benefit of an on-site blog would be that we can use the blog pages as potential landing pages for Adwords and other PPC marketing (sure we can do that with an off-site blog but it would require another click before they get to your main site). I think you could write your on-site blog posts in a way that would make the main site an informational resource for its theme. I think the SEO benefit would be better as a
result.

Keep in mind that Google hires teams of people to visit every site in their search index and rate them. The purpose is to improve the quality of search results and get spammy looking sites out of top rankings. I believe there are so many spammy looking blogs out there that just link out to other sites that their effect is decreasing over time as a result of this manual rating. The blogs like this are labeled as thin sites and devalued in ranking once they are reviewed. I think its better to focus on getting blog content on-site that makes your main site look more like an information resource. I would shy away from the traditional blog look and feel and try to make it look more like a rich resource of information about issues related to the topic. When it gets manually reviewed, you’ll more likely get a bigger thumbs up than you’d get from a thin off-site blog.

Of course this is all just an educated guess at best so take it all with a grain of salt.

I’ve been reading a lot of news lately about this Yahoo / Microsoft deal that fell through. Now the latest is that some big share holders are trying to force the deal to go through by replacing Yahoo’s board. It just goes to show how Wall Street misunderstands the situation. Stocks increased on the run up to the deal deadline. After it fell through, Yahoo stocks slipped. Anyone with half a brain would have picked up a few shares after the slip which is exactly what I did. Of course I’m not a 5% stake holder in the company but a few shares I do own and I felt like venting my frustration at the other vocal shareholders trying to force the deal with Microsoft.

Yahoo has long been in battle with Microsoft even from the early days with Hotmail. A deal with Microsoft would essentially devalue the Yahoo brand and have it lose respect among the geek community. Over the past year or two Yahoo has been growing its web business in the right direction. Its acquisitions of Flickr and Del.icio.us are just 2 examples off the top of my head. Those two names alone boost Yahoo’s appeal as a company which is why Microsoft is after them in the first place.

More importantly is their search technology. Despite Google’s dominance in search, Yahoo’s search engine has gotten extremely better over the past few years. Its results are typically much more accurate than Google’s in my opinion and they’ve been growing more and more appealing as an alternative to Google’s mythical magic search engine. Anyone who watches search engine rankings will tell you that Yahoo’s search results are much more consistent over time which makes it even more appealing. Not only that but SEO for Yahoo versus Google is more scientific and less voodoo. Its just another reason why Yahoo has been looking like its positioning itself more and more to become an extremely formidable online presence. Microsoft recognizes that and wants to low ball them before they secure their footing and get a steal on a buyout bid. The stockholders are on Microsoft’s side because they see a potential short term gain in stock value.

What they fail to see is that a Yahoo - Microsoft merger would essentially ruin any chances of a truly successful online presence for Yahoo and Microsoft. Microsoft has proven itself incapable of bringing anything worthwhile to the online arena even with past acquisitions of online businesses. Its ultimately a dooms day deal for Yahoo that’s being pushed by a few big stockholders who want some quick cash and have little interest in the long term success of Yahoo’s technology. I’m behind their technology and support them but selling out to Microsoft is likely going to sour the support Yahoo’s built up in recent years and ultimately fail as usual in the hands of Microsoft. Microsoft is a has been. Ask any kid who’s used a Mac before. They’re desperate and see the writing on the wall. They want to prolong their demise by purchasing promising web companies.

My guess is that Microsoft has wooed these stockholders privately in a lobbying type effort. Their thinking is that instead of raising their bid that they’ll woo a few big stockholders to replace Yahoo’s board and get their low ball deal for the company. Its typical Microsoft at its best. Either way, Yahoo’s stock is destined to go up whether they sell or don’t sell. No sale means they continue their forward progress and continue to build their momentum in a positive direction. I’ll likely keep my shares in that case. Sell and I’ll likely sell my shares shortly before the acquisition so I don’t end up with a loss at the end of the day. I’m no day trader but those are my thoughts as a web developer who’s seen these two companies have their better days.

I’ve been working with one client for several months now and they are extremely pleased with the results I’ve produced for them thus far. Here is a snapshot of the monthly analytics data for the site over the past year. You can get a larger view of the graph if you click on the image.

I started working with them in July and by August you can see that their traffic more than doubled. From July to now, I’ve continued to work with the client on increasing rankings through seo, link building, and improving visitor retention. The result is that their natural organic search traffic has seen an increase almost 10 times what it was, from 1,800 to 17,000. I’m pointing this out because I like the latest Google Analytics monthly graph view. It really helps in showing trends over time and its encouraging for clients to see the upward results from my seo work.

I’ve been through the ropes when it comes to SEO and PPC advertising (particularly with Google’s Adwords). Clients come to me either looking for SEO or online marketing solutions. Their objective is obviously to increase their sales and they want it to happen yesterday. The first thing I look at is where a site is currently ranked in search results for their targeted keywords before beginning any SEO or PPC campaigns. I like to also take a look at their Google Analytics data to see what’s already working well for them and to get a perspective of the volume of traffic they’re dealing with. Some business have none of these things setup and are simply trying to make a push to increase online revenue. Whatever their motivation the objectives are all the same and the same techniques apply. Except, however, for my recommendation for PPC ads.

Small businesses which don’t get much online traffic are usually looking for a quick fix and instant gratification. I usually take the time to explain the benefit of SEO and link building to these companies but in the end they usually choose PPC advertising. I’m not against PPC ads but I do have a pretty quantitative reason for trying to dissuade some of my clients from pursuing that route. From my experience, PPC ads only work if you have a high return on a product that has a high conversion rate per click. In essence a high ROI. The problem for many small businesses is that they sell low dollar items which themselves produce a small profit margin.

Lets take a small handmade soap manufacturer as a perfect example of a small business looking to expand their sources of revenue by setting up a shopping cart. A company like this might make a dollar on each bar of specialty soap sold. They aren’t likely to have huge resources like a large technology corporation so their budget for the project is relatively small. Lets say less than 5k USD. Part of that covers development costs and the rest ends up in limbo while I exude the benefits of SEO over PPC advertising to the client. From the perspective of a business if someone tells me to spend a few thousand dollars and it’ll pay off in maybe 6 months but they can’t guarantee first page placement my natural business sense tells me to tell this person where to go. My alternative is to pay per click advertising which will result in instant traffic and likely increase my sales immediately. The choice is simple in that regard and its why many small businesses with little understanding of online commerce end up abandoning their get rich quick online schemes.

Lets dig a little deeper into PPC ads. Lets assume a bar of my handmade soap costs me $3 and my typical order contains 3 bars of soap. Lets also assume I make $1.50 off each bar of soap I sell. Those numbers are pretty realistic examples. I now want to start an online marketing campaign and put in 3k USD to PPC ads for a month. That means I need to sell 2000 bars of soap in a month and have about 666 orders with 3 bars each on average. If we stretch those orders out over the entire month then I need to get about 22 online orders a day to simply break even with my PPC advertising campaign. Now lets look at the budget I have allocated for the month, $3,000. That’s about $100 a day. Lets assume that with my amazing skills I’ve managed to optimized my Adwords campaign so well that I’m getting an average CPC at $0.40. That will give me about 250 visits a day just from my Google Adwords campaign. If 22 of those 250 visitors placed orders that would make my conversion rate 8.8%. Depending on the industry that’s a pretty high conversion rate for online sales. Now take a look at what we just went over. This looks at what we need to just break even and we had to make some incredible assumptions like our amazing CPC at $0.40. A average CPC of $0.60 to $1.20 is more realistic for the targeted keywords but we gave ourselves the benefit of the doubt for this example. We still needed a 8.8% conversion rate which isn’t too probable. Its an example where some simple upfront analysis of the numbers will tell you that my $3,000 is better spent doing some SEO and link building so that I can get natural free search traffic that will last longer than a single month.

Lets take an example of a high dollar product, travel. People spend about $1500 on a travel sale and per sale I may make on average 10% or $150. The example above needs to be modified to have an average CPC at $2.00. If I need to make $100 a day in sales it means I need to make just 1 sale a day or 2 every 3 days. If I’m spending $100 a day on ads with an average CPC at $2.00 I’ll get about 50 visitors a day. If I need just 1 sale from those 50 visitors that’s a conversion rate of 2% which is certainly much easier to get than the 8.8% I needed from selling soap. This campaign has a high likelihood of succeeding and will probably be well in the black at the end of the month.

Travel has a high profit margin and requires a low conversion rate which is why PPC campaigns work for it. Soap has low profit margins and requires a high conversion rate to simply break even so a PPC campaign isn’t likely to succeed. My soap business will end up negative at the end of the month and I’ll likely be bitter at the waste of the small investment I made towards PPC ads.

In summary, SEO is the way to go for small businesses selling products with small profit margins. Its a more calculated and well thought out method of generating sales online. The benefits may take time to show but the payoff is that your online business may actually succeed. PPC ads on the other hand will have you bleeding money with little chance of ever turning a profit.

The other day I noticed a pretty significant SEO related problem with using a built in Rails construct. I noticed a problem when I started getting application errors that were letting me know that the user agent was none other than our friendly Googlebot. A closer look at the app shed light on a problem you may not have even expected. When using

it will take a look at the HTTP_REFERRER and redirect the user to that url. The problem, however, is that Googlebot doesn’t send a referrer and neither do a whole bunch of other search engine spiders. The result is that when they visit your site they get a nice 502 server error because Rails raises an exception. It doesn’t know what url to redirect to so it send a 502 error. Googlebot then sees your site as a bunch of 502 errors in the situations where you’re using redirect_to :back. Take a look at the Rails API and you’ll see the last line clearly mentions this.

The solution is to catch the RedirectBackError the redirect_to raises when there’s no referrer. Its a simple fix but one you need to be on the lookout for or else you might end up with a few 502 errors giving you bad mojo with the Google gods.

I was surprised to come up empty handed when searching for a Google Base Rails plugin. I wanted something that would allow me to easily create a feed into Google Base using their API. I didn’t find anything so I took the quick and short term solution and created my own Google Base xml feed. Its based on RSS 2.0 so its not incredibly difficult but I could have saved a few minutes if it was already written for me so here it is. My Google Base xml generator in Ruby on Rails. Its not complete and only has the fields that I specifically wanted for my products. Your feed will likely contain other fields so check the Google Base docs for more information on customizing it. You’ll notice that I thought Google Base was going to pull my xml feed when I initially wrote this but it turns out I have to use the API and this is just good for generating the xml file which you then have to manually upload to Google Base.

First, I added this to route.rb

Then I created controllers/google_controller.rb

And finally, I create views/google/base_feed.rxml

There are obviously calls to helper methods in the base_feed.rxml file like product_url and photo_url. I use those so I can easily generate pretty seo urls anywhere I need them. You’ll need to replace those with however you create your urls.

This should suffice for at most 31 days when all the products I just added will expire in Google Base. I doubt I’ll bother creating a Google Base Rails plugin unless I see a noticeable increase in traffic and sales so don’t hold your breath.

For those of you paying attention to the latest changes Google has been making you may be wondering what’s been going on. Some have speculated that they’ve been cracking down on paid link exchanges for high page rank sites. That may be part of the change but it certainly isn’t all of it. Besides updating page ranks they’ve also modified the weight they give to external links. For a while they were giving huge weight to good internal linking structures so everyone starting having really good internal linking. Now they’ve shifted their focus again to increase the weight of external links to your site. This might not be a direct algorithm change but could be a side effect of a page rank update. I’m thinking that as the page ranks become stale the weights on external links diminishes because they are less certain of the reliability of those external links. When the do a page rank update those external links count more. A natural side effect of a page rank update. They count more because they just updated them so they’re much more valid than they will be four to five months from now prior to another pr update. So if your ranking dropped as a result of the latest changes (but PR stayed the same) then I suggest you focus on building some better external links into your site. If your ranking increased, don’t just sit pretty and smile at yourself because you’re getting more traffic. You need to start solidifying your position by creating more content and continue working on your internal linking strategy. If your ranking pretty much stayed the same then you need a magic 8 ball because I have no answers for you.

On a side note, from my analysis of current traffic and ranking on many different sites it appears as if Yahoo’s rankings have also been adjusted. This is less than a week after Google’s latest update. It seems odd to me that Yahoo rankings are adjusted along with Google’s. I’ve seen sites who’s page rank go up yet Google traffic go down while at the same time Yahoo traffic increases. Could this be some sort of link between Yahoo rankings and Google rankings? What purpose would Yahoo have in learning from Google ranks? Well, if they had a method to their madness then certainly they could combine learning from Google ranks with what they believe to be good and bad sites to improve their own algorithm. Search engines are always trying to improve and filter out spammy sites. Yahoo does a much better job than Google when it comes to this. Their index is slow but steady. Google freaks out over every minor change you make to your site. Put a page up and a day later you take it down? Google flips out and tosses you into the 404 trash bin of junk sites. Yahoo is much calmer and collective when it comes to their index. It takes longer to get into it but when you are in you don’t have to worry about Yahoo freaking out because your site has a 404 or two every now and then. Its one of the biggest things that upsets me about Google’s index. The web is dynamic not static. If I put a page up today and take it down tomorrow it doesn’t mean my site is junk or isn’t worthy of high rankings. It means I’m adaptable. Some things work. Some things don’t. That’s how you develop a good site. You find out what works. With Google index though, if you publish something you better be damn sure that you want it to be online forever or else.

Another note, the proper way to remove a page from the net without Google pissing on you is to first remove all links to the page. Wait a few weeks. Even though some people claim Google updates their index continuously it isn’t true. They cache pages and base their indexing off of those cached pages. You need to wait weeks before Google goes through and updates all the cached pages it has for your site. It may continuously update its index based on a tiny subset of its index (say your main page to get the latest blog) but if you remove a deeply linked page it will take a while for it to work its way out of the system. Next, you’ll want to submit a url removal request with Google using their webmaster tools. Finally, once you’ve waited long enough you can remove the page. This procedure is completely ridiculous by modern standards. Google has some serious catching up to do with other search engines in my opinion. Their indexing is inherently flawed and their results are littered with junk spammy sites. Oddly enough as far as search results go Yahoo has much more relevant results and has a much more reliable algorithm than Google could ever hope for. Google needs to take a lesson from Yahoo on search. They seem to have focused on everything but search since their IPO. Its time to get back to your roots Google and fix the problems with your search that we’ve known for some time. They’re lagging and leaving the door completely open for a rival to move in. Someone just needs a better algorithm and enough momentum.

I’ve reached my limit of frustration with current keyword trackers. The technology is simple enough that it baffles my mind why so many keyword rank trackers are for profit. There doesn’t seem to be a single decent instance of an open source keyword tracker out there that I could find. I want something open and that can run in Linux of course but my searches have left me empty handed.

I’ve started designing my own keyword tracker as a result. I will release it under the GPL because I like to keep it real like that. It will be a Rails application and I will host a version for people to use free of charge (with some limitations so it doesn’t kill my servers). Basically, you can extend the app by creating a Rails plugin for it for different search engines. I will just write one for Google for starters. Hopefully I can get some community support to get more search engines working for it. I’ve got the database mostly planned out and will be starting the project in the next week or two. I will make an instance of Trac to help the collaboration and issue tracking.

Basically, I’ll be creating something that will have multiple users. A user can login and enter a new site or track an existing site. Each site has a set of keywords which the app tracks over time. I want graphs of the keyword activity over time and I want the ability to import keywords and export the rank history. If anyone is interested in helping me out on this project just comment on this post to let me know and I’ll set you up with a Trac account so we can get started.

Google punishes sites heavily for 404 errors. By the time you realize your site has an error its usually too late and you’re already being punished for them. I suggest you stay proactive on your 404 errors and use a link checker. I found this extremely useful tool. Xenu’s Link Sleuth. It basically crawls your entire site for every single internal and external link. You can chose to ignore external links if you want as well and just focus on internal links. It even visits images and mailto and just about anything that has a ’src’ or ‘href’ in the html of your site. I considered it a nice toy when I first used it but when it quickly found numerous serious 404 issues on a few of my sites I upgraded the importance of this tool in my toolbox. This will keep you ahead of the curve instead of constantly playing catchup. Try it out on your site and I guarantee you’ll find 404s you had no idea existed. It shows you all the sites that have the bad links on them as well so you know where to go to correct the problem. Best of all this software is free! Its a Windows application unfortunately but any self respecting web developer has virtual machines with different operating systems on them, Windows being one of them, so that shouldn’t be much of a problem if you’re a serious developer.