Why PeopleRank is Intrinsically Flawed
January 27, 2011 § 3 Comments
By: Xianhang Zhang
The Quora team talks a lot about experts & the power of expertise, and one of Quora’s explicit goals is to attract and retain more experts, empowering them to contribute high quality content on the site. At the heart of this initiative is the shiny new “PeopleRank” system supposedly in the works to algorithmically determine the precise degree of expertise each person contains. I’m here to argue that such an approach is fundamentally flawed in that, no matter how good the algorithm may be, it will not produce the desired results of attracting experts because its processing of garbage input can only product garbage output.
We see two parallel seams running through much of the cultural landscape: that of pop culture and that of expert culture. There are the blockbuster movies which are judged on their box office and art house movies which are judged on how good they are as movies. Academic journal publishing which cares about how true something is vs. science news reporting which cares about how interesting something is. The Michelin guide vs the Zagat Guide… the tension between expertise and pop culture runs deep within our social landscape.
These two cultures exist, side by side, because they represent fundamentally different and irreconcilable ways of looking at the world. Pop culture, as the name suggests, is literally popular culture. It’s the ultimate democratic system in which every dollar has an equal vote. Popular culture is consensus culture, where 2 + 2 can equal 5 if enough people believe it to be true. Expert culture, on the other hand, strives to be an objective culture. It strives to answer the much more tricky question of “is this true?” and “is this good?’ rather than simply “is this popular?”.
Ultimately, the problem with PeopleRank as a measure of expertise is that it’s actually not possible to do. Expertise can not be generated organically from the bottom up; only pop culture can be. What PeopleRank lacks is what computer scientists call “ground truth.” Ground truth is the platonic, authoritative version of the world in which your algorithm can be judged against. If you want a machine learning algorithm to understand the emotional expressions from people’s faces, you first need to hire a bunch of interns to manually classify each face to form your ground truth. Without such ground truth, such an algorithm becomes impossible to develop because you don’t even know what you’re developing for.
Similarly, PeopleRank can only work as an expertise ranking tool if it has accurate data of who is and is not an expert. For Adam & Charlie (Quora’s co-founders), this is possible within the realm of startups and technology because they are genuine experts in that area. They have some idealized version of what Quora should look like for startups and they can tweak the algorithm until it starts to look like that. But for everything outside of that area, Adam and Charlie know no more about what Quora should look than any random person. Sure, there are people on the site who would know what Quora should look like for with regards to e.g. Cooking, but this just moves the problem one step back in asking Adam & Charlie to pick who is good at picking who is good at writing answers. This recursive descent can only be terminated by Adam & Charlie exhibiting taste and, if they do not possess it, it means that PeopleRank can only discover expertise to the extent that Adam & Charlie can discover expertise; anything additional that it discovers is pop culture, not expertise.
So, what’s wrong with Quora as pop culture, you may ask? I mean, Quora has been pop culture ever since its beginning and it was fine then. The problem is Quora’s ambitions of scale. Pop culture is the culture of the community that it resides in and finds its own level within that community. The early community of Quora was highbrow and so was the culture. The problem is that pop culture exerts a steady gravity on quality which is pointless to battle. It’s pointless not because it’s impossible, but because it’s actually undesirable.
To better understand the fate of Quora, it is instructive to look at the story of the Food Network over the span of its existence. The Food Network, in its infancy, was a channel devoted to the serious examination of food and cooking. High quality shows featuring respected chefs producing good content was their primary staple. It may have been pop culture but it was good pop culture. Over the years of its existence, the Food Network has engaged in a steady and relentless process of dumbing down. Gone are the chefs, to be replaced by a bunch of knowledgeless housewives. Gone is the cooking, replaced by ditzy housewives and eating contests. In short, the Food Network became crap. It’s useless to theorize about how to prevent this process because the truth is that the correct course of action was to become crap: coupled with the rise to crapdom for the Food Network were steadily increasing viewership figures and amazing financial results. Remember, in pop culture, popular is all that counts. (For the much more in-depth story, I recommend Bill Buford’s New Yorker article).
What happened to the Food Network has happened to every niche cable station from the History Channel to the SciFi channel to MTV. To live in the world of pop culture means that you have to play by its rules. There are plenty of people who refuse to believe this and maintain their artistic integrity and purity while still trying to play in this market. With only a few exceptions, we call these people “failures.”
Quora needs to resolve its identity crisis if it is to move forward. It fortuitously managed to attract a small core of high quality people to start with and mistakenly took its highbrow pop culture for expertise and has laid that down as a strategic plank in their business strategy. By opening the service up to the world, it now needs to decide whether it’s going to accept its pop culture DNA or unsuccessfully reach for the glory of its early days.
Xianhang Zhang is a frequent contributor and oft-downvoted agitator on Quora. He weeps for the loss of high-brow cooking shows on cable networks.