Faking Data for a Good Cause
Americans don't always trust scientific research (or so studies say). Sometimes they're absolutely right to be suspicious, as recent incidents concerning reproducibility, philosophy and advocacy masquerading as science, and the politicization of science, demonstrate. Questionable studies say stories don't help either.
Today I get to tell you about an experiment that is going above and beyond the normal call of science to double-check itself for accuracy.
The Laser Interferometer Gravitational-Wave Observatory--LIGO--refreshingly avoids the trap of cutesy acronyms such as SMART or AMAZING. LIGO is a project only one step below the famous Large Hadron Collider in size and difficulty. The main facilities consist of four ultra high vacuum tubes; each tube is over two-and-a-half miles long. Exquisitely sensitive interferometers within the tubes measure spacetime, hoping to see it grow or shrink by a factor of roughly 10-21 (one billionth of one billionth of one percent, or a total distortion a thousand times smaller than a proton) as a gravitational wave ripples through. The mission and the incredible precision of the facility in pursuing this work are worthy of an entirely separate account.
Scientific details aside, I was fascinated to read about a method, effective but also brutal, that LIGO uses to check itself for scientific accuracy. It's called blind injection.
A thousand or so scientists work on LIGO. Out of this group a handful are chosen for a special mission: the injection team. They are allowed to manipulate the raw data from the LIGO instruments. They may choose to hide fake detection signals in that data without telling anyone else. The rest of the project operates entirely in the dark. Unaware, the whole collaboration may find the signal and believe that they are on the verge of a scientific breakthrough that will earn them a Nobel Prize. They may also realize that the detection is a fake or else fail to catch it at all.
The fake is introduced in a particularly smart way: the injection team is allowed to directly wiggle the mirrors in the detector to imitate the movement caused by a true gravity wave flowing through the detector. Hence, the only way to know if a signal really came from black holes smashing together millions of light years away is to finish an entire scientific study of the event and then ask the injection team to reveal whether the data was faked.
This investigation carries out a real-world test of the facility. Can the enormous and complex analysis, carried out by hundreds of scientists all over the world, correctly identify the expected signal of a real gravitational wave? That's a very important question: an enormous experiment looking for something that has never been seen has no a priori operator's manual.
A fake signal can carefully calibrate the expectations of the operators as well. It forces them to run every single instrument through every single possible test to see if it could have accidentally introduced spurious noise. After witnessing the faster-than-light neutrino debacle of two years ago, a collaboration knows how vitally important it is to check for every single possible source of error, no matter how small. (That mistake was eventually traced to a loose cable between two machines.) Scientists must conduct extensive investigations of miniscule possibilities along the lines of: Could a stray puff of air have bumped the side of the tube? Could we have our own loose cable somewhere?
Blind injection also lines up the ability of the instrument to correlate its result with other methods of detection. Did other telescopes of different types see further evidence for the right kind of event happening in the right place in the sky? A black hole merger also emits a massive flash of x-rays, for example.
Does the blind injection idea work in practice?
When a potential gravitational wave signal was detected on the instrument in September of 2010, the entire LIGO team went to work. Through six months of late nights, they looked at every last detail and wrote up scientific reports of their finding. When the paper was complete, a mass vote was held to decide whether to submit that paper to the eager scientific journals of the world. The vote was a unanimous yes. Dramatically, the team responsible for injecting events dramatically opened an envelope to reveal whether the event was real or an injected fake.
The envelope opened in March 2011 to reveal a fake. The good news was that the team correctly identified the signal. The better news was shown by comparing the final study results to what the injection team had expected them to see. There were two discrepancies. Both of these turned out to be mistakes by the blind injection team themselves, revealed by the sweat of the blind LIGO team!
There is one brutal coda to this process: experimenters work very long hours at projects like LIGO. When a massive discovery is looming on the horizon, they squeeze extra work into nights, weekends, and extra shifts away from their homes and families. Tests like this run the risk of burning out researchers who sacrifice everything only to find out that their work was for naught. The blind injection method is great science but it's extremely hard on individual scientists.
Tough it may be, but this sort of rigor is extremely pertinent in the current climate of science. Poor methods, biased intentions, and reproducibility problems undermine everyone's confidence in scientific studies. The researchers at LIGO go above and beyond to try to insure utmost accuracy and care in their results.