About The Gazette Search Back Issues Contact Us    
The newspaper of The Johns Hopkins University December 4, 2006 | Vol. 36 No. 13
Scientists 'Turbocharge' Star Data to Zoom Into First-Place Finish

By Lisa De Nike

A team of experts from Johns Hopkins and the University of Illinois at Chicago's National Center for Data Mining recently won the seventh annual Bandwidth Challenge, held in Tampa at SC06, the international conference for high-performance computing, networking and storage.

On Nov. 16, the team transported 1.3 TB of data from the University of Illinois at Chicago to the SC06 floor in Tampa with a sustained data transfer rate of 8 Gb/s over a 10 Gb/s link and a peak rate of 9.18 Gb/s: That's the equivalent of transmitting 2000 CDs worth of data more than 1,000 miles in less than 20 minutes, according to team member Alexander Szalay of Johns Hopkins.

"Two or three years ago, the only way we were able to distribute such large data sets was to ship three whole computers around the world by Federal Express," said Szalay, Alumni Centennial Professor in the Henry A. Rowland Department of Physics and Astronomy at Johns Hopkins. "Using the 800 kbits/sec DSL connection that many of us have at home, a transfer of a data set this large would take almost six months, and using the more typical upload speed of 384 kbits/sec, it would take almost a year. Moving data at such speeds opens up whole new ways of approaching scientific problems."

The data set involved in the competition was from the Sloan Digital Sky Survey and, when compressed, consisted of 60 files of about 23 GB each, totaling 1.3 TB.

The technology that made the transfer possible was an open source high-performance network transport protocol called UDT that the NCDM developed several years ago. Since then it has been downloaded more than 8,000 times, using an open source peer-to-peer storage system called Sector, also developed by NCDM. Sector was built using UDT and was designed to distribute large e-science data sets. The team then used a 10 GB/s network called PacketNet, which was provided by National Lambda Rail.

In the past, UDT and technologies could move data at high speeds but faced challenges when utilized to transport data from disk to disk over long distances, according to Szalay, who said that Sector allowed the team to transport large data sets from disk to disk quite easily.

"During the challenge, we copied the data over in 20 minutes from disks in Chicago to disks in Florida, ready to be used immediately," he explained. "Previously, we would have had to ship disks around, which is a cumbersome process involving a lot of work on either end."

The Johns Hopkins team — Szalay, Ani Thakar, Jan vandenBerg and Alainna Wonders — and the researchers at University of Illinois would eventually like to use such network speeds as part of their everyday research.

Michael McCarty, chief network officer for Johns Hopkins, said he believes that such collaboration could lead to benefits for not only Johns Hopkins researchers but the community at large.

"We have been excited to have an opportunity to work with Alex and other scientists across Hopkins whose research creates new challenges for us as well. Today, we have partnerships with the state of Maryland, several Maryland universities and the Internet 2 community that we have been able to leverage to help make efforts such as this possible," McCarty said.


The Gazette | The Johns Hopkins University | Suite 540 | 901 S. Bond St. | Baltimore, MD 21231 | 443-287-9900 |