Success Factors in Transformative Research

BOSTON—One of the final sessions of the 2013 AAAS Annual Meeting here looked at transformative research and the factors that facilitate it—or don’t.

The kinds of informal things that scientists do matter for scientific identity and the production of transformative knowledge in ways that people really don’t appreciate.

Speaker Terttu Luukkonen, a research and innovation policy scientist at the Research Institute of the Finnish Economy in Helsinki, explained that transformative research—which in Europe is better known as groundbreaking, cutting-edge, or frontier research—can take many forms. It could be a new way of seeing things, a methodological or instrumental advance, the discovery of a new substance such as graphene, or an emerging subfield. Serendipity also plays a role, Luukkonen added.

Luukkonen focused her talk on the ERC peer review process and its ability to select transformative research proposals. Based on an ongoing analysis of interviews with ERC panelists, Luukkonen and her colleagues found that selection panels deeply understand what is expected of them in terms of promoting both excellence and novelty. But the greatest debates during selection often occur with proposals that are in the middle of the pile, where controversial and unconventional proposals tend to end up, Luukkonen said. She identified several ways in which ERC panels try to reach agreement that may play against transformative research proposals. When voting is used to decide a proposal’s fate, support tends to favor more conventional and less risky proposals, she said. Another way that panels try to achieve consensus is by assessing risk: trying to sort the feasible from the nonfeasible by looking at the researcher’s capacity, equipment availability, and research tradition.

ERC panels would like to promote research that is highly transformative, but they are also “very concerned with risk-taking,” Luukkonen said. “Although it is aimed to support really novel ideas, [ERC] has certain constraints because it is a public body.” Private charities and foundations have more freedom to support unconventional ideas or focus evaluation on the people rather than the research proposals, she explained. The ERC system, which has awarded more than 3000 grants since 2007 and evaluates both people and research plans, appears to be a compromise. But “whether the ERC succeeds or not, we will only be able to see in 20 years’ time,” Luukkonen concluded.

Sociologist John N. Parker of Arizona State University, Tempe, next examined the collaborative environments that enable transformative research, offering the National Center for Ecological Analysis and Synthesis (NCEAS) in Santa Barbara, California, as an example. NCEAS identifies general patterns and principles in ecology by looking at data from a cross-disciplinary perspective. One way that NCEAS does so is by hosting working groups of 10 to 15 people from various disciplines in which they can focus on analyzing and synthesizing information for 1 or 2 weeks at a time. “NCEAS is an incubator, or an organization that facilitates transformative social movements in science,” Parker said. It played a crucial role in catalyzing the formation of ecophylogenetics as an emerging new subdiscipline, for example.

Looking at the social processes that enable transformative research within NCEAS working groups, Parker and his colleagues found that generating trust, solidarity, and group identity among people who often begin as total strangers is important. Allowing the group to get to know one another through social outings helps team members build trust and organize their individual roles and tasks. “The kinds of informal things that scientists do matter for scientific identity and the production of transformative knowledge in ways that people really don’t appreciate,” he said.

Age structure within groups is also important. “There is a lot of talk about are junior scientists more transformative in their work or older scientists? What we see is that, really, it’s the blend of the two that matters,” Parker said. He and his colleagues found that, within NCEAS groups, “on one hand senior scientists provide socialization, networking, institutional memory, orientation in the field, and junior scientists [bring] fresh ideas, new techniques, labor, energy, they ask challenging questions, they push the senior scientists to new ways of thinking,” he said. “This balance between junior and senior really matters for production of transformative science.”

He also identified several processes important for creativity.

  1. Peer review on the fly: Groups are able to evaluate and build on ideas at a speed that could not be reached otherwise.

Innovation and science metrics researcher Bhavya Lal of the Institute for Defense Analyses’ Science and Technology Policy Institute in Washington, D.C., next looked at the myths and realities of transformative research. Based on a review of the literature and funding programs pertaining to transformative research, Lal and her colleagues developed several hypotheses about what made research transformative. She then tested them by comparing the recipients of a high-risk, high-reward NIH program with top researchers funded by NIH’s mainstream program.

Lal’s research, which has been submitted for publication, suggests that contrary to common belief, transformative researchers are no more prolific in publishing than other excellent researchers. She also found that the common perception that transformative research is risky and associated with a high rate of failure in the long term doesn’t hold up: The perceived risk of failure at the time of the award was uncorrelated with the impact or innovativeness of the research as assessed by experts 5 years later. She did find, however, that the popular hunch that transformative research tends to rely on innovative approaches is true, in that the researchers used unproven or difficult techniques, or used existing techniques creatively.

Many programs have been created to support transformative research, yet “the challenge is that there is very little insight in what characterizes potentially transformative research,” Lal said. Right now, these programs are based on intuitions and rules of thumb, she added. To examine these intuitions analytically could help design better programs, improve peer review, and develop metrics to evaluate these programs. Right now, the dominant paradigm for a definition of transformative research is, “I know it when I see it,” she said.

The Romance of Science

Gaining Independence