Doing what we do best as SEOs

[dazzlindonna]

Wow, a reporter from the NY Times was allowed inside the lair of the Google search engineers and he actually managed to get some pretty incredible tidbits from the head search dude, Amit Singhal.

We, as SEOs, thrive on taking the tiniest hints of information and expanding our knowledge from them. So I've taken the meatiest portions of the article linked above, and separated them out here into sections, so that we can chew on, think about, digest, absorb, and extrapolate from each. Enjoy. (Things in bold and capitalized are mine).
************************
Amit Singhal said, “Search over the last few years has moved from ‘Give me what I typed’ to ‘Give me what I want.’”

“Someone brings a query that is broken to Amit, and he treasures it and cherishes it and tries to figure out how to fix the algorithm,” says Matt Cutts.

THIS IS A RECENT ALGO CHANGE:
Some complaints involve simple flaws that need to be fixed right away. Recently, a search for “French Revolution” returned too many sites about the recent French presidential election campaign — in which candidates opined on various policy revolutions — rather than the ouster of King Louis XVI. A search-engine tweak gave more weight to pages with phrases like “French Revolution” rather than pages that simply had both words. (WOW, I WONDER IF SOME OF THE RECENT CHANGES I'VE SEEN HAVE TO DO WITH THIS).

THIS IS AN OLDER ALGO CHANGE:
At other times, complaints highlight more complex problems. In 2005, Bill Brougher, a Google product manager, complained that typing the phrase “teak patio Palo Alto” didn’t return a local store called the Teak Patio.

So Mr. Singhal fired up one of Google’s prized and closely guarded internal programs, called Debug, which shows how its computers evaluate each query and each Web page. He discovered that Theteakpatio.com did not show up because Google’s formulas were not giving enough importance to links from other sites about Palo Alto. (EARLPEARL AND LOCAL AFICIONADOS, TAKE NOTE OF THIS).

It was also a clue to a bigger problem. Finding local businesses is important to users, but Google often has to rely on only a handful of sites for clues about which businesses are best. Within two months of Mr. Brougher’s complaint, Mr. Singhal’s group had written a new mathematical formula to handle queries for hometown shops.

INTERESTING, NEW AND UPCOMING:
But Mr. Singhal often doesn’t rush to fix everything he hears about, because each change can affect the rankings of many sites. “You can’t just react on the first complaint,” he says. “You let things simmer.”

So he monitors complaints on his white board, prioritizing them if they keep coming back. For much of the second half of last year, one of the recurring items was “freshness.”

Freshness, which describes how many recently created or changed pages are included in a search result, is at the center of a constant debate in search: Is it better to provide new information or to display pages that have stood the test of time and are more likely to be of higher quality? Until now, Google has preferred pages old enough to attract others to link to them.

But last year, Mr. Singhal started to worry that Google’s balance was off. When the company introduced its new stock quotation service, a search for “Google Finance” couldn’t find it. After monitoring similar problems, he assembled a team of three engineers to figure out what to do about them. (NOW SEE, THIS KINDA TICKS ME OFF. WHEN IT HITS HOME, THEY DO SOMETHING ABOUT IT. WE BYTCHED FOR HOW LONG ABOUT SANDBOX PROBLEMS?)

Earlier this spring, he brought his squad’s findings to Mr. Manber’s weekly gathering of top search-quality engineers who review major projects. At the meeting, a dozen people sat around a large table, another dozen sprawled on red couches, and two more beamed in from New York via video conference, their images projected on a large screen. Most were men, and many were tapping away on laptops. One of the New Yorkers munched on cake.

Mr. Singhal introduced the freshness problem, explaining that simply changing formulas to display more new pages results in lower-quality searches much of the time. He then unveiled his team’s solution: a mathematical model that tries to determine when users want new information and when they don’t. (And yes, like all Google initiatives, it had a name: QDF, for “query deserves freshness.”) (NOTE THE NEW TERM WE HAVE NOW.)

Mr. Manber’s group questioned QDF’s formula and how it could be deployed. At the end of the meeting, Mr. Singhal said he expected to begin testing it on Google users in one of the company’s data centers within two weeks. An engineer wondered whether that was too ambitious.

“What do you take us for, slackers?” Mr. Singhal responded with a rebellious smile.

THE QDF solution revolves around determining whether a topic is “hot.” If news sites or blog posts are actively writing about a topic, the model figures that it is one for which users are more likely to want current information. The model also examines Google’s own stream of billions of search queries, which Mr. Singhal believes is an even better monitor of global enthusiasm about a particular subject. (SO, IF YOU WANT YOUR BRAND NEW SITE TO AVOID THE SANDBOX, MAKE SURE IT FITS IN WITH A HOT TOPIC).

ANOTHER TIDBIT:
Google recently developed a new system that can hold far more data and search through it far faster than the company could before. (I WONDER WHAT "RECENTLY" MEANS...).

REGARDING SIGNALS:
Mr. Singhal has developed a far more elaborate system for ranking pages, which involves more than 200 types of information, or what Google calls “signals.” PageRank is but one signal. Some signals are on Web pages — like words, links, images and so on. Some are drawn from the history of how pages have changed over time. Some signals are data patterns uncovered in the trillions of searches that Google has handled over the years.

“The data we have is pushing the state of the art,” Mr. Singhal says. “We see all the links going to a page, how the content is changing on the page over time.”

TERMS AND FINAL RANKINGS:
Once Google corrals its myriad signals, it feeds them into formulas it calls classifiers that try to infer useful information about the type of search, in order to send the user to the most helpful pages. Classifiers can tell, for example, whether someone is searching for a product to buy, or for information about a place, a company or a person. Google recently developed a new classifier to identify names of people who aren’t famous. Another identifies brand names. (ALWAYS NICE TO HAVE OFFICIAL TERMS TO USE...SIGNALS AND CLASSIFIERS).

These signals and classifiers calculate several key measures of a page’s relevance, including one it calls “topicality” — a measure of how the topic of a page relates to the broad category of the user’s query. A page about President Bush’s speech about Darfur last week at the White House, for example, would rank high in topicality for “Darfur,” less so for “George Bush” and even less for “White House.” Google combines all these measures into a final relevancy score. (AND ANOTHER ONE...TOPICALITY)

The sites with the 10 highest scores win the coveted spots on the first search page, unless a final check shows that there is not enough “diversity” in the results. (HMM, VERY INTERESTING) “If you have a lot of different perspectives on one page, often that is more helpful than if the page is dominated by one perspective,” Mr. Cutts says. “If someone types a product, for example, maybe you want a blog review of it, a manufacturer’s page, a place to buy it or a comparison shopping site.”

As an example, he points out what happens when cities suffer power failures. “When there is a blackout in New York, the first articles appear in 15 minutes; we get queries in two seconds,” he says.

Mr. Singhal says he tested QDF for a simple application: deciding whether to include a few news headlines among regular results when people do searches for topics with high QDF scores. Although Google already has a different system for including headlines on some search pages, QDF offered more sophisticated results, putting the headlines at the top of the page for some queries, and putting them in the middle or at the bottom for others. (I ASSUME THIS ALSO ENDED UP TYING IN WITH UNIVERSAL SEARCH).
*******************
So, there you have it, folks. Let's discuss!

[kbf]

(SO, IF YOU WANT YOUR BRAND NEW SITE TO AVOID THE SANDBOX, MAKE SURE IT FITS IN WITH A HOT TOPIC).

that explains it.....

nice post

kf

[dazzlindonna]

Of course, if your brand new site DOESN'T fit in with a hot topic, then you'll have to MAKE it a hot topic.

[SEO_AM]

Donna... Thank you for your post. It fills a few holes in my thought patterns. Kudos!

[pops]

QDF, signals, classifiers, topicality, diversity (in results)

I'm dying to see the ideas you guys come up with to take advantage of these concepts. And, of course, any ideas you have on how to make a topic "hot."

[skitzzo]

Wow... I'm impressed Google let so much information out to be honest. I'm still digesting but Donna, thanks for starting this thread. I think there's a LOT we can either confirm or extrapolate from this. I especially like the "hot topic" info. I've got to go read everything again but I'm excited.

[pittbug]

Great post Donna, thanks very much for sharing. Looks like we were a little short on our list of signals

Local search:
theteakpatio.com - shame he didn't also mention how dupe content works with its other site: regencyteak.com

Great to hear more feedback on getting local (geo-focussed) links, as opposed to just content themed links.

Topicality: strange - this seems like it looks at onpage factors... I wonder if this is a new thing, or whether it was just not publicly mentioned before....

Top 10 Diversity:
This explains why wikipedia is so prominent, and also explains that it can be all but impossible to get into the top 10 for some terms, unless you're the authority on the topic. Do a search for [ipod] and you'll see that the top 10 are not all stores. A good indicator of the classifier at work is if you now search for [ipod store].

QDF: This is hard to force, since the news sources will surely be the big media outlets. But, it does confirm something that was posted in the big patent document. Not all queries need to bring back the freshest content. Some pages will rank well for some terms even if they haven't been updated in months/years.

[kbf]

Originally Posted by dazzlindonna

Of course, if your brand new site DOESN'T fit in with a hot topic, then you'll have to MAKE it a hot topic.

right, and on that note, I've found that you can definitely punch some semi competitive queries on a new domain into the top ten just by brute force of overwhelming amount of unique interlinked text content...... (sort of like making it hot) ....... that's my favorite way .... it was working exceptionally well during last autumn and into the new year, and, i thought g was getting things right but it seems they turned the knob a little the other direction in around march or so of this year, but, i'm still getting success off it, but, a little more difficult: i have a feeling though it's going to revert back to how it was last autumn ..
kf

[eKstreme]

Thanks for this Donna. I'm still digesting it. I have to say, the hotness of a page/topic really affects rankings. When my Googlebot requested a CSS file went popular on Digg and got picked up around the web, it shot to number 1 in the G SERPs for many keywords. As time went by and the web moved on, the rankings slowly decreased. Now it ranks only for a very narrow set of keywords.

Pierre

[A.N.Onym]

I've been noticing the 'freshness boost' for years now. Lately, it's been more noticeable, though, it seems.

Though some claim that this article reveals a lot, I sincerely doubt it reveals anything new that an attentive webmaster or SEO didn't know.