By common consensus of some 20 investigators contacted for this story from within and without ADNI, this behemoth study is working well overall. It is on plan, having enrolled all subjects. It is calling some participants for their year 3 follow-up visit, and its cylinders are cranking on data cleanup and analysis. Importantly, scientists generally applaud the standardization of procedures it has achieved across sites. But not everything went smoothly. “There have been some corrections and tune-ups along the way,” said Clifford Jack of the Mayo Clinic in Rochester, Minnesota, who heads ADNI’s MRI core and oversees quality control of the MRI images. “At a global level things are working well, but the closer you get to a large operation, the more chaotic it is.”

Snafus in the execution of a huge study offer learning opportunities for future studies. For example, one additional distributed consortium study modeled on ADNI, called DIAN, is gearing up and in its own planning has looked closely to ADNI. Beyond DIAN, ADNI has generated widespread interest in big-science studies. According to Andrew Saykin of Indiana University in Indianapolis, the ADNI model of data standardization and cooperation has come up even outside of Alzheimer disease in discussions of neuroimaging studies of psychiatric diseases, brain injury, and cancer. In this spirit, then, what could be done better next time?

“In retrospect, we should have allocated more funds to support the quality control process,” said Neil Buckholtz of the National Institute on Aging. “In terms of its basic structure and spirit of cooperation ADNI has done terrifically. But just tracking who is to get what procedure and that people come in at the right time, then making sure the data comes in at the time that it should, that it comes in the required form—those sorts of issues take more effort and person hours than we thought.”

A related issue is that the decision to make the imaging data publicly available almost instantly has subsequently raised questions about its quality. The concern is that if the data on LONI is raw, not processed, individual investigators who download it will have to clean it up themselves before analyzing it and introduce variation in this way. Some sites do their own reprocessing of ADNI data; for example, Nick Fox’s group in London uses differential bias correction (Lewis and Fox, 2004) on ADNI scans before measuring brain atrophy using their boundary shift integral method. That said, the ADNI imaging core teams do perform some preprocessing. The MRI imaging data come in two flavors, said Jack. The images become publicly available worldwide in their raw form as soon as a participating site uploads them. Also at that time, Jack’s team pulls those raw images to the Mayo Clinic, where scientists check their quality, correct errors, and follow up with the uploading site where they encounter a problem. “This is the data we encourage investigators to use,” Jack said.

What are those problems? Frequently they have to do with upgrades and subsequent software changes that have been performed at a local scanner unbeknownst to the ADNI scientists there, and become apparent through scan variations during ADNI quality control. Of the 84 scanners in ADNI, most are in clinical use at their respective hospitals, and ADNI scans take up less than a percent of their use, Jack said. Or sometimes the scans are acquired in the wrong way, and the participant has to return to the site for a re-scan. Another source of error can be the ADNI phantoms. These are a fleet of 66 identical water-filled objects that support calibration and preprocessing. They must be scanned along with a person’s MRI scan at every site. Other scientists noted privately that the phantoms had not performed quite as well as hoped for given their expense in terms of purchasing and scan time. Part of the problem is that they are not completely identical, and they can introduce a confound when one is dropped and has to be replaced. Some variation in the quality of the MRI scans arises from the different scanner models, and ADNI will address which model and field strength is the most suitable for ADNI’s goals. Given 20-20 hindsight, Fox and others noted that they might have spent more time in the preparatory phase to test scan sequences and test the value of the phantoms.

An ADNI participant who had an MCI diagnosis at study entry and again 18 months later (top images) showed relative lack of atrophy (view larger image) compared to a participant who initially was diagnosed with MCI but progressed to a diagnosis of AD 18 months later (view larger image). Image credit: Clifford Jack, Mayo Clinic, Rochester, Minnesota

ADNI’s biomarker study had its own share of “oops” moments while fine-tuning the operating procedures for this complex undertaking. Headed jointly by Les Shaw and John Trojanowski at the University of Pennsylvania Medical School in Philadelphia, the biomarker core personnel receive a dozen or so packages every day from the 58 participating sites, which send in urine and plasma samples (56 of those send in CSF, as well). “We expect by the end of the study to have received enough samples to create more than a quarter of a million aliquots of biofluid,” Shaw said. Every box has to be received, logged into the core database, thawed, aliquoted, and frozen into what’s casually called the “ADNI freezer farm” at UPenn. This, too, works well overall.

One of the bigger mishaps was that from the beginning of the study through March of 2007, some sites collected CSF samples into polystyrene tubes. Aβ sticks to this plastic, and use of those tubes has been shown to falsely lower CSF Aβ measurement by up to a third, according to Cindy Carlsson at the University of Wisconsin ADNI site. Carlsson, a geriatrician who conducts trials using CSF markers as outcome measures, discovered that the sterile CSF collection tubes in the ADNI lumbar puncture kits sent through a contract research organization were made of polystyrene. ADNI had provided each site with the polypropylene tubes needed for storing Aβ properly, but they were not sterile; hence, sites using a CSF drip technique that requires sterile tubes ended up using the wrong plastic. Carlsson added that ADNI fixed this problem as soon as it became apparent. When asked how many samples were affected, Shaw e-mailed to ARF: “We studied this very carefully, including by directly comparing CSF collected in polypropylene tubes to CSF collected in polystyrene tubes that are included in the standard lumbar puncture tray, and using an exposure time that most closely mimicked that for ADNI collections. We saw only minor decreases in Aβ42 and tau, none in P-tau, and hardly detectable [change] in the tau/Aβ42 ratio. This important issue was not a likely contributor to significant variability in our study.”

A second hiccup occurred around shipping logistics. It highlights the need for planning operating procedures down to minute details in such a large and complex study. The 58 local ADNI sites ship blood samples to the National Cell Repository for Alzheimer’s Disease (NCRAD), where the staff establishes cell lines for genetic studies on ADNI participants. According to NCRAD's principal coordinator Kelley Faber, ADNI samples often arrived at NCRAD on Saturdays, when NCRAD is closed. This might have been because participating sites ship some samples to NCRAD and others to the biomarker core at UPenn, which does receive shipments on Saturdays. On Saturdays, the ADNI boxes destined for NCRAD then got dropped off at a different lab at Indiana University. Not being part of the cell repository, that lab often refrigerated the samples, which is detrimental to the growth of the cell lines, Faber wrote to ARF. The NCRAD staff repeatedly instructed ADNI sites to hold samples drawn on Friday at room temperature and ship them on Monday, and also asked their local colleagues to store the Saturday arrivals correctly until Monday. The problem occurs less often now but, due to staff turnover in all those different labs, has not gone away. Were samples lost? “I don’t think many, if any, were lost, as we tried to give them extra TLC,” Faber wrote.

Moreover, samples sometimes arrive unaccompanied by their requisite form. This can be a major issue because the NCRAD staff then cannot easily double-check identifying information on the tubes. At other times, urine or CSF arrived at NCRAD instead of UPenn and had to be forwarded there. Overall, however, Faber noted that ADNI is going well. “Everyone was always friendly and willing to work together. The main issue was due to the number of sites and people working on the project. A lot has been learned from the ADNI study and will be used to improve future studies,” Faber concluded. Tatiana Foroud of NCRAD concurred that ADNI is going smoothly given its size and complexity. “We were very pleased to be a part of the ADNI project,” said Foroud.

Several scientists interviewed for this article regretted the timing of ADNI’s PIB-PET substudy. When the original ADNI grant was written in response to the request for applications the NIA had issued in October 2003, amyloid imaging was generating buzz at conferences and initial human data had been published (Engler et al., First human study with a benzothiazole amyloid-imaging agent in Alzheimer’s disease and control subjects. Neurobiol Aging. 2002;23 suppl 1:S429). But PIB was not formally ready for prime time, because the first widely noticed paper appeared in March 2004 (Klunk et al., 2004).

By the time ADNI’s amyloid imaging substudy was funded and set to go, ADNI had already enrolled so many participants that the substudy had to accept people who had undergone baseline FDG and MRI scans a year before, according to Chet Mathis of University of Pittsburgh School of Medicine. This cut into its ability to obtain three consecutive longitudinal scans on normal control and MCI volunteers, and to compare that side-by-side with ADNI’s other assessments. “We wish we had started sooner,” Mathis wrote to ARF. On the upside, the study made up the lost scans by enrolling more people than originally planned.

Finally, some scientists privately shared a concern that frequently accompanies big distributed science consortia beyond AD research. While they expect that ADNI will achieve its primary goals, they also say that it will necessarily do so at a low common denominator in terms of data quality. This concerns, for example, imaging resolution. That is because ADNI, despite everyone’s best efforts at standardization, must still absorb a considerable degree of remaining site variation. A related concern voiced privately is that ADNI may create the impression that the big problems are solved. In the process, it absorbs a large chunk of the available funding at the expense of more technically innovative, cutting-edge imaging and biomarker research in individual labs that might pay off later. That said, all sources agreed that the field has to have multisite data to move forward.

In most interviews, talk about fumbles and compromises came as part of a larger sense that overall, ADNI has worked surprisingly well. The problems underscore how daunting a task it has been to coordinate such a large undertaking, and how many obstacles had to be overcome. Here’s Sterling Johnson, co-principal investigator with Sanjay Asthana of the University of Wisconsin site. “There has been a high degree of planning, thoroughness, and standardization on every aspect of the complex data collection protocol.”—Gabrielle Strobel.

This is Part 5 of a six-part series. See also Parts 1, 2, 3, 4, 6. Read the entire series [.pdf].