Institute For The Future

Measuring How Well We Work Together

During the recent Breakthroughs to Cures game, which we ran in collaboration with the Myelin Repair Foundation, one of the key themes that emerged was that basic science research would be aided by a lot more collaboration. While this sounds nice in theory, one of the challenges is that it's easy to evaluate researchers by the number of papers they have published, but it's much harder to measure collaboration, much less base tenure and promotions and the like on how well on works with others.

 

Which was why I was surprised but happy to see a recent article in Nature about new efforts to use network analysis to gauge how likely a researcher is to collaborate with others.

 

"In our own institution, we have institutional resources and they get directed to these centres, but we don't measure their output," [John Hogenesch] explains. "We're making investment decisions without really having a formal description about how we measure success."

 

He hopes that his team's new paper in Science Translational Medicine may eventually be developed into a way of assessing collaboration. By quantifying the number of papers published and grants obtained by ITMAT researchers, the team produced a 'network analysis' of the relationships between them. Hogenesch likens it to a cocktail party.

 

You might choose to measure how good a party is by looking at how many people talk to each other, he says. For example, if two people speak to each other for longer than five minutes, they would have what the researchers call an "edge" — a connection between them on a network analysis. More edges mean a better party.

 

"The actual edges in our graph are papers and grant applications," says Hogenesch. And the edges or connections between researchers represent their collaborations.

 

 

 

Now, at some level, I'm not sure I buy the method--nor do I think that a second method outlined in Nature, to measure collaboration based on shared publications, truly captures what it means to work well with others. Shared publications and grants and so on are examples of a certain kind of collaboration--one that is long-standing, successful, and involved, though, people can learn a lot from each other from a chance meeting, hour-long conversation, or some other engagement that is far less involved. It's a mistake, in other words, to assume the only interactions worth measuring are the ones that take an enormous amount of work.

 

For that matter, it's probably a mistake to assume that the ability to play nicely with others is equivalent to the ability to be good at collaborating. As Tim Hartford recently noted:

 

The risks of committee thinking were highlighted in 1972 by the psychologist Irving Janis in his famous analysis of the role of “groupthink” in the Bay of Pigs fiasco. Groupthink is the tendency of committees to congeal around a particular point of view, reassured by the fact that everybody agrees with everybody else, and nervous about expressing dissent.

 

A more abstract demonstration, if a powerful one, was delivered in the 1950s by the psychologist Solomon Asch. Asch showed his experimental subjects four lines and asked them to say which two of the four were of equal length. When the hapless subjects were surrounded by actors pretending to be doing the same task, and blatantly delivering the wrong answer, many of the real experimental subjects fell in line with the group, and expressed clear signs of stress as they did so....

 

Asch’s experiments showed that if there was a single dissenter in the room, the experimental subject was far more likely to resist social pressure and pick the correct pair of lines. This was true even if the dissenter himself was also wrong. What mattered was that he said something different.

 

 

 

I think there are a couple broader points here--which often seem to pop up any time a researcher tries to begin measuring something new--the first of which is that we don't have good measures for variables like collaboration precisely because they're slippery concepts that mean different things to different people. If it were easy to measure well-being, for example, we would never have started using GDP as a proxy for well-being in the first place. We use GDP because it's a lot easier to agree on a measure of economic output than it is to agree on a measure of happiness.

 

Collaboration is the same sort of thing--since one's contribution to a group depends on the dynamics of the group as a whole, it seems unlikely to me that any single metric could capture what we want to know about collaborative abilities. In other words, to measure collaboration, we don't simply need scores for individuals--we need better tools to understand how individuals fit into groups. It's easy to imagine even more granular ratings--Researcher A, say, turns out to excel at thoughtful, provocative dissent in order to advance discussion, but is lousy at creating consensus, whereas Researcher B has the reverse scores.

 

The second point which often seems to come up with these new measurements is this: Just because something is really hard to measure doesn't mean that we shouldn't try to measure it. Our measurements, like GDP, carry real weight--and so if we ignore variables like collaboration simply because they're hard to understand, we do so at our peril.

 

This, again, has been true in the research world--where numbers of publications, because they are tangible and easy to measure, take the place of less obvious but potentially more critical information.

 

My final point has to do with how we relate to measurement in general. We like to think of measurements, and numbers in general, as carrying a level of precision. But as we move into trying to measure more abstract concepts, I think we'll also need to learn to think of these measurements as, at least in many cases, best guesses rather than rigorous pieces of data.