Google Head of Search Spam Matt Cutts posted a videotoday answering how Google goes about evaluating which new search algorithms they use and which they throw away or adapt.
The question was posed by James Foster of Sydney, Australia who asked:
What are some of the metrics that Google uses to evaluate whether one iteration of the ranking algorithm is delivering better quality results to users than another?
Matt Cutts breaks it down to about three steps of the evaluation process:
(1) They test the algorithm offline, benchmarking how the results rank with the new algorithms and if the URLs are higher quality than the previous algorithms in place. The quality is based on how the search quality raters rate the URLs in previous cases. If the URLs were unrated, Google can request these raters to rate the new URLs or compare the old search results to this new test set. Then based on those metrics, Google may decide to move the test to the next phase.
(2) Live tests, where Google will sample a subset of real live searchers and give them the new results with the new set of test algorithms. If Google sees a higher click rate on the new search results, it may imply that the new results are better than the older ones. This is not always the case, specifically with webspam, Cutts said. But in general, the more clicks on a specific search result page, the better quality the results.
(3) Then the Google Search Quality Launch Committee has the ultimate say on if the algorithm goes live to the public or not.
Matt said Google has this down to a “pretty good system” but every now and then they need to refine some of the processes within this workflow.
http://searchengineland.com/googles-matt-cutts-evaluate-new-search-algorithms-188044
