I read an interesting article from the Wall Street Journal entitled “On the Internet, Everyone’s a Critic, But They’re Not Very Critcial“. In it, Geoffrey A. Fowler and Joseph De Avila note that the “average grade for things online is about 4.3 stars out of five”. This may be a surprise to many (or at least it was to me) as a good part of the criticism I hear about web - especially social web - is in regard to the capability and aptitude of people to write/speak negatively. We hear about the ability of one negative experience-turned-review to snowball into a train wreck begging for disaster relief (see Pete Blackshaw’s Tell 3000).

Grade inflation (Positive Acquiescence Bias - thanks Bryce

download Imagine That

) seems to be prevalent across the web - YouTube and Amazon are both reporting it, and averages are higher in the UK (4.4) than the US.

While I don’t see any critical problems with positive acquiescence bias on the public web (buyer/browser/analyst beware & get smart), it made me think about the manifestation of this behavior on internal social networks - especially as more organizations are exploring and deploying mechanisms and/or re-engineering for some degree of social business design (Alimeter Group, Dachis Group).

Significant capital outlay for technology, change management, HR moves and more means that social business design is an investment. The investment is worth it, according to McKinsey survey results, but as companies push deeper into the space and begin to rely more heavily on information and insights gleaned from digital environments, I think we need to be aware of patterns and possible skews.

Why?

Because connecting a workforce is proving valuable: real-time feedback and data mean fast learning, course correction and innovation. Reputational systems applied to knowledge, resources, and options can quickly gauge a global and disparate organization’s sentiment and needs, allowing for informed business decisions… ‘informed’ being the operable word here. If feedback is inflated (one way or another), organizations need to be wary of making decisions based on it. The opportunity to gather and act on data is certainly there. It’s the qualification of that data, per the tendencies being reported in similar environments, that must be remembered.

So…

As Brian Link says, “sample sizes and % participation and correlated results from different data sets are key to interpreting these kinds of things” but I’m also wondering - Do we design against/for it? Do we coach against it? I’m not even close to an expert on reputational systems (again, see Bryce)) and haven’t researched inflation results beyond the WSJ article, but it made me think:

1. Are grades inflated?
2. Are there reputational system design considerations which can be made to combat or normalize this behavior or the data?
3. Is there group behavior coaching or leadership that can modify this these patterns?

Thoughts?

Thanks for the shout-out, Nick! And - hoo-boy, you’ve opened a can of worms so late in the evening. No time to even begin to address some of the issues you raise. Other than to say… here’s a great series on systems for collective choice by Christopher Allen (with some real recommendations for improving their efficacy): http://www.lifewithalacrity.com/2005/12/systems_for_col.html and, of course, we’re currently trying to pull together a book that will hopefully address some of these issues. You can read our entire draft chapters at http://buildingreputation.com/

This is interesting stuff. I think simply being aware of this inflation is half the battle. As a business monitoring reviews and ratings of products and services online, the real insight can come from comparison to past online results. So compare inflated ratings to inflated ratings over time.

I have not taken time to read the article that caught your attention, but I’d be interested to know how using one’s real name as a username vs. more of an anonymous username affects positivity/negativity online. Are people more likely to be critical of products, brands and each other when they are posting as SweetSneakers03 rather than a name they go by in every day life? I would think so. So as social web leads us to developing an online identity that is more closely tied to our true identity, we are more positive with our interactions. Maybe.

I believe this problem is much more widespread than just web. It seems that even in physical interaction we tend to overlook small shortcomings in products and services when giving feedback unless the fault is so significant that we are willing to rally the troops against it. How often has a restaurant manager stopped by your table to ask how things are, and you reply “fine” although the bread was cold, your drink was flat, and the waiter has the personality of a rock?

In order to change this behavior, we need to change our attitudes towards honest constructive feedback as a society, not just as those designing UI for web. We too often ask people what they think only as bait for a glowing review that will feed our ego rather than to truly solicit honest suggestions to improve our product, service, or self.

In the meantime, maybe anonymity is the best short-term solution in these internal corporate environments where people’s jobs depend on a positive peer and employer image. Anonymous feedback would probably have much fewer trolls and flamers in a corporate intranet than elsewhere with web, allowing for honest feedback with reduced fear of personal attack for opinion.

I try to be honest with those who ask for feedback, but they must be willing to really listen and process constructive criticism before I will waste my breath giving it. Too often, our personal relationships are fractured when we give honest feedback that people really aren’t willing to hear.

I tend to agree with Kevin that this is not only a problem online but online is where it is most evident.
Here are a few thoughts to consider:
* In youth sports today, everyone is given a trophy (regardless of wins & losses)
* Many youth sports programs do not even keep score during the game
* In schools there are 3-4 different “honor rolls” where you find more kids on them than not
* In MBA classes I teach, the class applauds for students when they present their findings from a group discussion

I guess what I am saying is that the “self-esteem” movement has ruined us. Back in the 1970’s there were studies done to show that students’ self-esteem was correlated to their grades. This meant we could not hurt their feelings or make them feel inferior in any way or it would scar them for life?!? This has all been dis-proven but the damage was done and continues to this day.

Not only can people not leverage feedback of any kind, they also expect to be rewarded for simply showing up and doing the minimum expected. That somehow merits praise? This is a pervasive issue within our society and it will take effort and fortitude to address.

Great post!

I agree with Kevin — an element of anonymity is key to authenticity. As soon as a person has to be accountable (to whatever degree) for an opinion or comment, they’re going to be somewhat reserved and not completely forthcoming.

It’s human nature to be liked especially in a workplace where you have to see the same faces every day. So why rock the boat? Negative comments will likely be crafted and spun in ways so feelings aren’t hurt, thereby (possibly) compromising accuracy and timeliness.

So while a system may require my name to be listed by my comments and opinions, some things need to be done anonymously.

Finally, I also think a passive, logic based system needs to be used as well. If certain people’s opinions are viewed more often, there’s nuanced intelligence to be gleaned. Maybe that person is naturally respected more in an organization. That should factor into somehow/someway; maybe that person’s votes are pro-rated a bit more when they weigh in. I like the idea of systems that identify and reward value vs. just meaningless participation.

Just my random 2 cents…

Hey Nick, love the website. I need one of these!

On the post, I actually sent it out internally to my company’s marketing dept and VP’s. I got very similar responses to the comments here.

I think people are reading really far into the psychology of this.

The real question to me is: SHOULD one compare ratings of different things to each other e.g. printer paper to boots OR youtube videos to hard goods. Also, SHOULD one feel comfortable with any X.X/5.0 score without something VERY similar to benchmark against.

To accurately take value from ratings one must: Compare them to something VERY similar OR compare them to the same rating sometime in the past.

If you do this, you remove all the noise that everyone seems to be so focused on i.e. because the same noise exists in both.

This is the best way to leverage ratings because you will never have a true rating of whatever it is you are trying to measure. Consumers can’t rate well, consumers only rank. So a rating will never be hard and fast because people suck at it (myself included).

The value in ratings is being able to monitor how they change over time i.e. being able to learn what behaviors causes them to go up or down.

I think people are asking too much of the data here. If you don’t ask too much of the data, there’s really no problem.

thoughts?

I think that from a business end, ratings aren’t actionable. Asking someone “Do you like A or B” isn’t as relevant as showing one group A and one group B and getting numbers on which makes more sales (or preforms the desired metric)

I guess another way of that is: The question isn’t “do you like this?” The question is “Would you put your money/time/attention on this?”

Grades are inflated, recommendations are inflated, ratings are inflated, words are inflated. Personally, actions are the only accurate measurement in this realm.

It happens everywhere: YouTube, Yelp, etc. deal with the same problem.

http://youtube-global.blogspot.com/2009/09/five-stars-dominate-ratings.html