Beware of weak correlative scores

Written by

Warning: this content is older than 365 days. It may be out of date and no longer relevant.

In the World of Warcraft, there exists one number that can make or break your day, depending on who you’re interacting with: GearScore. GearScore is a mathematical formula that tries to rank players based on what equipment their character has, on the assumption that harder to get equipment means you’re a better player for having it, much in the same way that driving an expensive car might indicate more personal wealth. People looking to organize groups in the game often recruit for their groups solely by advertising GearScore requirements: “Looking for damage dealers, 5K GS minimum!”. Anyone who doesn’t meet this score doesn’t get invited to the group.

(WIN) Moriturus, 80 Death Knight — WTF is my Gear Score?

(FAIL) Krystos, 80 Paladin — WTF is my Gear Score?

Funny, both characters are the same player behind the keyboard…

The problem with GearScore is that harder to obtain gear isn’t necessarily indicative of a more skilled player. At best, it’s a weak correlation. For example, a player that works primarily in a healing role can get a very high GearScore from wearing damage dealing equipment – but that player will be completely ineffective as a healer. A player can have one character that is supremely well equipped but might have a second character that he just created that will have an abysmally low GearScore. The player behind the character may be incredibly talented, but the equipment and thus the GearScore will not reflect this fact.

Why do Warcraft players looking to create groups rely on such a potentially unreliable scoring mechanism? Because in the absence of better metrics, it’s what they’ve got to work with for making snap decisions, and the weak correlation is still strong enough that on average, a group composed of high GearScore players is somewhat more likely to fare better against fire-breathing dragons than a group composed of low GearScore players.

So what does a geeky algorithm like GearScore have to do with anything? For years, companies, especially in financial services, have evaluated potential employees based on credit scores. Like GearScore, credit score may have some correlation to a future employee’s abilities to be effective, but given how tumultuous the economy has been in the last 3 years, any company relying on this number may lose perfectly good candidates.

Why would a company rely on such a mechanism? For the same reason the Warcraft folks do – it’s a metric that lets computers and/or HR clerks filter through piles of resumes very quickly. Set a minimum credit score of 700 and your job as an HR clerk is much easier, as you’ll throw away 80% of the resumes in your inbox immediately.

So what if you don’t work in financial services? What if you’re a social media person instead? Surely no one would try to boil down the complexities of managing mass human interactions into a single number. Well…

Twitter / Michelle Tripp: Blow your mind? In some co ...

Is there more to you than this one-dimensional metric? Probably. Will people push this score or another like it just like the Warcraft folks push GearScore? Probably. Be prepared to address it if you’re a social media professional, because there’s an ever-growing chance that a decision-maker may hire or pass on you in an instant based on this one number.

Comments

15 responses to “Beware of weak correlative scores”

August 22, 2010

Eric Boggs

Great post, Chris. I’m not a big fan of “indexed-single-number-that-encapsulates-everything” marketing metrics for many of the reasons that you’ve described, primarily oversimplification. Don’t even get me started on “averages” as an indicator…

Reply
August 22, 2010

LynetteRadio

Not a fan of the scoring systems. What one person/company values is different than others, and homogenizing the system isn't helping anyone. Net Promoter scores are a farce as far as I'm concerned also. How many people that fill out the “how likely are you to recommend” 1-10 survey really put detailed thought into clicking a radio button? A decent starting point maybe – but have a brain and do your own research. Don't be lazy.

Reply
August 22, 2010

Bob Strachan

I agree that you can’t sum the complexities of person in a single score, and to hire a person based solely on their Klout (or Gear!) score would likely not end well… 🙂 but would you agree that as an intelligent metric of twitter influence Klout is a solid algorithm? If I was hiring a community manger or social media marketing ‘guru’, I’d look at their Klout score to see if they practice what they preach. Would you recommend Klout as a way to assess a medium/long term Twitter campaign for a brand? Are there better or supplemental metrics you’d include?

Reply
August 22, 2010

Joe Fernandez

Hey Chris,

I am one of the founders and the CEO here at Klout. We actually hear from a lot of HR folks that they are using the Klout score as an input in filtering for certain positions. If you are hiring for a community manager or marketing role this makes a ton of sense and is a huge step up in sophistication then just looking at follower count. Also, with the Klout score we provide HR manager (or anyone) the ability to really dive into the details of how a person leverages social media to spread their message. You can see historical trends on all the components of the score and gain insight into a person’s communication style. We think going beyond a single score and giving this kind of context is critical for decision makers.

We also hear that many people are starting to put their Klout score on their resume. I think people appreciate the fact that with Klout they can easily demonstrate that they have authority and are able to drive engagement around specific topics even if they don’t have a huge audience.

I think the bigger point you are trying to make here is important though. We all want to feel that we are judged on the merits of who we really are not on our Klout score, credit score, college or who our parents know. At the same time it’s funny how we all look for signals like this to segment other people. I think it’s something in our wiring. One of the things I am most proud of with Klout is that even though it is used sometimes to “judge” people, the score is totally in your control. If you consistently create content that others finding interesting and helpful you will have a high score. The person who is most influential about Rolls Royce could totally be a 16 year old who has never touched one in real life. I think it would be really difficult for companies to recognize these types of unsung heroes without the type of technology we are building.

Reply
August 23, 2010

Christopher S. Penn

Hey Joe,

I suppose it makes sense for certain very specific social media positions where you want to see if someone practices what they preach. That said, authority is kind of sticky. Two simple examples:

My Klout score is 51. It says I’m influential to a degree, a specialist, and I’ve got a string of cool badges next to my name. Yay me, right?

The Klout score of MassGovernor – Deval Patrick, the governor of the state – is 40. He’s less influential than me? I think not. One of the two of us can mobilize the National Guard, and it’s not me.

Now, I fully and wholly recognize that this is due to the limitations of the system, of the data that’s available, and that the governor of Massachusetts hopefully has better things to do than Tweet, like run the entire state, but what concerns me is that lazy people will believe that a Klout score or any scoring algorithm is an effective substitute for thinking and doing your homework.

By the way, my CEO would like a word with you when you have a moment 🙂

Thanks for commenting.

Reply
August 23, 2010

Christopher S. Penn

If it was someone I was serious about hiring, I’d focus more on interviewing them and understanding how they understand strategy and where the role of CM/social media practitioner fits in the big picture.

Simply put: your Klout score doesn’t tell me if you know how to make social media profitable/effective for my company.

Reply
August 23, 2010

Mckra1g

What I do find helpful about measurement indicators like Klout = the dispassionate aggregation that a computer/algorithm sees that we are blind to within ourselves.

The closest analogy I can come to is photography, especially black and white images. The camera “sees” what we do not sometimes. Vestiges of emotion on faces, body language, lighting subtleties….

Speaking to Joe’s point, having this information helps us to manage and cultivate areas we wish to strengthen.

Great post – made me think. My best, M.

Reply
August 23, 2010

Guest

That’s a shame, because my “Boggs Score” is like 104 or something. At least the median is.

Reply
August 23, 2010

Christopher S. Penn

That's a really good point and perspective, especially given the somewhat self-important opinions that some characters in social media have of themselves.

Reply
August 23, 2010

Jim Raffel

*gasp* you mean we can't replace “thinking and doing homework?” My kids will be heartbroken 😉

Seriously, you make excellent points Chris. Looking at any machine generated score without also being willing to look at reality is a recipe for disaster. In certain areas of “thinking” you are far more influential than the Mass Governor. You, for example, have a great deal more influence on me than he does at the moment.

Maybe part of the answer is that we never lose site of context. A “score” taken out of context is not only potentially meaningless it's also potentially dangerous.

(Mine's 60 by the way so that would say I'm more influential than you Chris and I am not good sir, I am not)

Reply
August 23, 2010

michelletripp

Hi Chris, This is such a great post. I had no idea World of Warcraft is using a Klout style score. The thing that's most interesting about that is how it encourages a “culture of influence” in consumers who aren't in the social media marketing world. Judging people based on a number will begin to feel more comfortable for the general population. It's something that doesn't look like it's going away so the best we can hope for is that talented guys like Joe Fernandez work really hard at making the Klout algorithm as comprehensive as possible.

According to them, they're currently only measuring Twitter engagement. That's supposedly going to be changing, but I find it a little alarming that HR and recruiters are hinging decisions on one single channel metric.

I have a lot of respect for Joe and his team, and after initially questioning the negative impact of Klout in a recent blog post, I've been forced to embrace the fact that this is just how things are evolving. I think the better decision is to encourage the product to be the best it can be and keep users educated on its limitations.

I've talked to Klout in depth and have another post in the works, including some interviews with companies regarding whether they'd use it for screening candidates. It's going to be interesting to see how pervasive this new trend actually is and how it will impact all of us.

Reply
August 23, 2010

Steve Ward

Even a Social Media Community Manager cannot have their effectiveness as a Community Manager measured by their personal Klout score. Personal use and work use are often separated and to overly automate the human process of recruitment and selection is a dangerous way to go.
Similarly with psychometric testing – sure, do it to enhance your interview questioning, but never, ever make a decision based on random preference selections and an outcome that can be different every day.

The real problem is that `trust` in recruitment and selection has dissipated to the point where interviews and references are not enough; we need to see influence scores, reputation monitoring, facebook pages, and the lottery of personality tests to help us make a decision. This is sad, and in my practical experience, makes selection decisions often cock-eyed.

Reply
August 23, 2010

Matt Ridings – Techguerilla

Nice post Chris. For me, the real challenge is that the trust/usage of scoring systems like Klout is gaining far faster than the actual technologies accuracy.

Klout is by far the most representative of the services I've used, yet still with massive flaws even within its contextual purpose (Turn off Twitter for a couple of weeks and re-run your score…see what happens. Did you really lose your influence because you went on vacation?). As savvy builds in the usage of these systems people will be able to selective use individual measures that makeup a score for applicable purposes, but in general people simply take the aggregated score at the moment which can lead to all sorts of errors in judgement.

Will Klout become a sort of Meyers-Brigg in the future? Perhaps. But let's hope it reaches that level of accuracy before that point happens 🙂

Reply
August 23, 2010

Chel Wolverton

Once someone gets the interview, in this job market, is anyone really going to argue that it’s not a good thing to judge people by professionally?

I’m guessing that Klout score doesn’t correlate to intelligence, most people will assign it the value that the company they are applying for is giving it rather than dismissing it. Nor does it mean that the applicant will disagree with a company policy during a first or second interview, perhaps even not after hiring.

So we have to give the Klout score attention because it’s it’s a one stop measurement to make the HR department lives easier? Don’t give it attention, stand up for your beliefs that it doesn’t matter and you might not make the interview cut at all.

Fun job market out there!

Reply
August 24, 2010

Batman

Actually, my Klout score is better than my credit score, so, in this case, it might be a good thing… 🙂

Reply