Why it’s next to impossible to prove that gun laws work. From 538.
So, did California’s gun laws succeed at making it one of the safest states … or did they fail to stop a string of mass shootings? Questions about the efficacy of gun laws have gotten easier to answer in recent years as changes to federal policy have helped to bring money and people back to the field of gun violence research. But decades of neglect mean there are still lots of blank spaces — policies that don’t yet have good quality data backing them up. A recent report from the Rand Corporation that reviewed the evidence behind a variety of gun policies found just three that were supported by evidence that met the report’s quality standards.1
That fact, however, doesn’t mean other gun laws don’t work — just that the research proving it doesn’t yet exist. Scientists I spoke to saw it as an “absence of evidence” problem, stemming from long-standing, intentional roadblocks in the path of gun violence research. Even the authors of the Rand report say lawmakers should still be putting policies aimed at preventing gun violence into practice now — regardless of what the science does or doesn’t say.
“I think that the goal of the lawmaker is to pick laws that they have a reasonable hope will be better than the status quo,” said Andrew Morral, a senior behavioral scientist at the Rand Corporation. “And there’s lots of ways of persuading oneself that that may be true, that don’t have to do with appealing to strict scientific evidence.”
California doesn’t just have some of the nation’s strictest gun laws and lowest gun death rates, it’s also maybe the best state to study gun laws in, said Dr. Garen Wintemute, director of the Violence Prevention Research Program at University of California, Davis Medical Center. That’s because of both the way the state makes data available to researchers and its willingness to work with researchers to further the science. Wintemute is currently part of a team that is working on a randomized controlled trial of one particular California gun law — an initiative that tracks legal gun owners over time and dispatches authorities to remove their weapons if those people later break a law or develop a condition that would make them ineligible to own guns in the state.
It’s hard to oversell what a big deal this is. Frequently referred to as the “gold standard” of evidence-based medicine, randomized controlled trials split participants randomly (natch) into groups of people who get the treatment and groups that don’t. Because of that, it’s easier for researchers to figure out if a medication is actually working — or if it just appears to be working because of some other factor the people in the study happen to share. These kinds of studies are crucial, but almost impossible to do with public policy because, after all, how often can you randomly apply a law?
Rand team found just three policies that have strong evidence supporting outcomes — and two of these are about the negative outcomes of policies that increase gun access. Stand-your-ground laws, which allow gun owners to use deadly force without trying to leave or deescalate a situation, appear to increase firearm homicides. Meanwhile, conceal-carry laws, which allow gun owners to carry a gun in public places, appear to increase the number of all homicides and increase the number of firearm homicides, specifically. The only laws restricting gun ownership that have this level of evidence behind them are child-access prevention laws, which have been shown to reduce firearm suicide, unintentional self-injuries and death, and homicides among young people.
That makes gun control laws seem flimsy, but it shouldn’t, Morral said. Instead, the lack of evidence ought to be understood as a product of political decisions that have taken the already challenging job of social science and made it even harder.
The Dickey Amendment, first attached to the 1996 omnibus spending bill famously prevented the Centers for Disease Control and Prevention from funding gun violence studies for decades. A new interpretation of that amendment in 2018 changed that, but Dickey wasn’t the only thing making it hard to study gun violence.
Instead, the researchers told me, the biggest impediment to demonstrating whether gun control policies work is the way politicians have intentionally blocked access to the data that would be necessary to do that research.
“So for instance, the federal government has this massive, great survey of behavioral risk indicators that they do every year in every state,” Morral said. “And you can get fantastic information on Americans’ fruit juice consumption as a risk factor for diabetes. But you can’t get whether or not they own guns.” Not knowing gun ownership rates at the state level makes it hard to evaluate causality of some gun control policies, he explained. “And it’s not because anyone thinks [gun ownership] is not a risk factor for various outcomes. It’s because it’s guns.”
The missing data problem also includes the 2003 Tiahrt Amendment that prevents the sharing of data tracing the origins of guns used in crimes with researchers, said Cassandra Crifasi, co-director of the Center for Gun Violence Solutions at Johns Hopkins University. “So now all we can see are these sort of aggregate-level state statistics,” she said. “We can no longer look at things like, when a gun is recovered in a crime, was the purchaser the same person who was in possession of the gun at the time of the crime?”
Recently, researchers have even been missing basic crime data that used to be reported by the FBI’s Uniform Crime Reporting program. Law enforcement agencies and states were supposed to be shifting to the relatively new, much more detailed National Incident-Based Reporting System, but the transition has been a catastrophe, with some of the biggest law enforcement agencies in the country not yet making the switch because of financial and logistical complications, Smart said. “The FBI has not been able to report for the last eight quarters whether homicide rates are up or down,” Morral added.
But much of the data that’s not available at a national level is available in California, Wintemute said. “Unlike researchers in any other state, we have access to individual firearm purchaser records,” he told me — the very data the Tiahrt Amendment blocks at the national level. “We do studies involving 100,000 gun purchasers, individually known to us, and we follow them forward in time to look for evidence of criminal activity or death or whatever the outcome might be that we’re studying,” Wintemute said.
Unfortunately, because the data is only available in California, the results of those studies would only be applicable to California — making it data that wouldn’t be considered high-quality in the Rand report. Wintemute can demonstrate if a policy is working in his home state, but not whether it works in a big, broad, existential sense. It wouldn’t count towards expanding the number of policies Rand has found evidence to support. This is something researchers like Crifasi see as a flaw in the Rand analysis, but it’s also a reason why Morral and Smart don’t think evidence-based policy is a good standard to apply to gun control to begin with.
It’s useful to know what there is evidence to support, Morral said. “But we don’t at all believe that legislation should rest on strong scientific evidence,” he said. Instead, the researchers from Rand described scientific evidence as a luxury that legislators don’t yet have.
“There’s always gonna be somebody who’s the first person to implement the law,” said Smart. “And they’re going to have to derive their decision based on theory and other considerations that are not empirical scientific evidence.”
So, did California’s gun laws succeed at making it one of the safest states … or did they fail to stop a string of mass shootings? Questions about the efficacy of gun laws have gotten easier to answer in recent years as changes to federal policy have helped to bring money and people back to the field of gun violence research. But decades of neglect mean there are still lots of blank spaces — policies that don’t yet have good quality data backing them up. A recent report from the Rand Corporation that reviewed the evidence behind a variety of gun policies found just three that were supported by evidence that met the report’s quality standards.1
That fact, however, doesn’t mean other gun laws don’t work — just that the research proving it doesn’t yet exist. Scientists I spoke to saw it as an “absence of evidence” problem, stemming from long-standing, intentional roadblocks in the path of gun violence research. Even the authors of the Rand report say lawmakers should still be putting policies aimed at preventing gun violence into practice now — regardless of what the science does or doesn’t say.
“I think that the goal of the lawmaker is to pick laws that they have a reasonable hope will be better than the status quo,” said Andrew Morral, a senior behavioral scientist at the Rand Corporation. “And there’s lots of ways of persuading oneself that that may be true, that don’t have to do with appealing to strict scientific evidence.”
California doesn’t just have some of the nation’s strictest gun laws and lowest gun death rates, it’s also maybe the best state to study gun laws in, said Dr. Garen Wintemute, director of the Violence Prevention Research Program at University of California, Davis Medical Center. That’s because of both the way the state makes data available to researchers and its willingness to work with researchers to further the science. Wintemute is currently part of a team that is working on a randomized controlled trial of one particular California gun law — an initiative that tracks legal gun owners over time and dispatches authorities to remove their weapons if those people later break a law or develop a condition that would make them ineligible to own guns in the state.
It’s hard to oversell what a big deal this is. Frequently referred to as the “gold standard” of evidence-based medicine, randomized controlled trials split participants randomly (natch) into groups of people who get the treatment and groups that don’t. Because of that, it’s easier for researchers to figure out if a medication is actually working — or if it just appears to be working because of some other factor the people in the study happen to share. These kinds of studies are crucial, but almost impossible to do with public policy because, after all, how often can you randomly apply a law?
Rand team found just three policies that have strong evidence supporting outcomes — and two of these are about the negative outcomes of policies that increase gun access. Stand-your-ground laws, which allow gun owners to use deadly force without trying to leave or deescalate a situation, appear to increase firearm homicides. Meanwhile, conceal-carry laws, which allow gun owners to carry a gun in public places, appear to increase the number of all homicides and increase the number of firearm homicides, specifically. The only laws restricting gun ownership that have this level of evidence behind them are child-access prevention laws, which have been shown to reduce firearm suicide, unintentional self-injuries and death, and homicides among young people.
That makes gun control laws seem flimsy, but it shouldn’t, Morral said. Instead, the lack of evidence ought to be understood as a product of political decisions that have taken the already challenging job of social science and made it even harder.
The Dickey Amendment, first attached to the 1996 omnibus spending bill famously prevented the Centers for Disease Control and Prevention from funding gun violence studies for decades. A new interpretation of that amendment in 2018 changed that, but Dickey wasn’t the only thing making it hard to study gun violence.
Instead, the researchers told me, the biggest impediment to demonstrating whether gun control policies work is the way politicians have intentionally blocked access to the data that would be necessary to do that research.
“So for instance, the federal government has this massive, great survey of behavioral risk indicators that they do every year in every state,” Morral said. “And you can get fantastic information on Americans’ fruit juice consumption as a risk factor for diabetes. But you can’t get whether or not they own guns.” Not knowing gun ownership rates at the state level makes it hard to evaluate causality of some gun control policies, he explained. “And it’s not because anyone thinks [gun ownership] is not a risk factor for various outcomes. It’s because it’s guns.”
The missing data problem also includes the 2003 Tiahrt Amendment that prevents the sharing of data tracing the origins of guns used in crimes with researchers, said Cassandra Crifasi, co-director of the Center for Gun Violence Solutions at Johns Hopkins University. “So now all we can see are these sort of aggregate-level state statistics,” she said. “We can no longer look at things like, when a gun is recovered in a crime, was the purchaser the same person who was in possession of the gun at the time of the crime?”
Recently, researchers have even been missing basic crime data that used to be reported by the FBI’s Uniform Crime Reporting program. Law enforcement agencies and states were supposed to be shifting to the relatively new, much more detailed National Incident-Based Reporting System, but the transition has been a catastrophe, with some of the biggest law enforcement agencies in the country not yet making the switch because of financial and logistical complications, Smart said. “The FBI has not been able to report for the last eight quarters whether homicide rates are up or down,” Morral added.
But much of the data that’s not available at a national level is available in California, Wintemute said. “Unlike researchers in any other state, we have access to individual firearm purchaser records,” he told me — the very data the Tiahrt Amendment blocks at the national level. “We do studies involving 100,000 gun purchasers, individually known to us, and we follow them forward in time to look for evidence of criminal activity or death or whatever the outcome might be that we’re studying,” Wintemute said.
Unfortunately, because the data is only available in California, the results of those studies would only be applicable to California — making it data that wouldn’t be considered high-quality in the Rand report. Wintemute can demonstrate if a policy is working in his home state, but not whether it works in a big, broad, existential sense. It wouldn’t count towards expanding the number of policies Rand has found evidence to support. This is something researchers like Crifasi see as a flaw in the Rand analysis, but it’s also a reason why Morral and Smart don’t think evidence-based policy is a good standard to apply to gun control to begin with.
It’s useful to know what there is evidence to support, Morral said. “But we don’t at all believe that legislation should rest on strong scientific evidence,” he said. Instead, the researchers from Rand described scientific evidence as a luxury that legislators don’t yet have.
“There’s always gonna be somebody who’s the first person to implement the law,” said Smart. “And they’re going to have to derive their decision based on theory and other considerations that are not empirical scientific evidence.”
