A team of experts from Johns Hopkins and the
University of Illinois at Chicago's National Center for
Data Mining recently won the seventh annual Bandwidth
Challenge, held in Tampa at SC06, the international
conference for high-performance computing, networking and
storage.
On Nov. 16, the team transported 1.3 TB of data from
the University of Illinois at Chicago to the SC06 floor in
Tampa with a sustained data transfer rate of 8 Gb/s over a
10 Gb/s link and a peak rate of 9.18 Gb/s: That's the
equivalent of transmitting 2000 CDs worth of data more than
1,000 miles in less than 20 minutes, according to team
member Alexander Szalay of Johns Hopkins.
"Two or three years ago, the only way we were able to
distribute such large data sets was to ship three whole
computers around the world by Federal Express," said
Szalay, Alumni Centennial Professor in the
Henry A. Rowland
Department of Physics and Astronomy at Johns Hopkins.
"Using the 800 kbits/sec DSL connection that many of us
have at home, a transfer of a data set this large would
take almost six months, and using the more typical upload
speed of 384 kbits/sec, it would take almost a year. Moving
data at such speeds opens up whole new ways of approaching
scientific problems."
The data set involved in the competition was from the
Sloan Digital Sky
Survey and, when compressed, consisted of 60 files of
about 23 GB each, totaling 1.3 TB.
The technology that made the transfer possible was an
open source high-performance network transport protocol
called UDT that the NCDM developed several years ago. Since
then it has been downloaded more than 8,000 times, using an
open source peer-to-peer storage system called Sector, also
developed by NCDM. Sector was built using UDT and was
designed to distribute large e-science data sets. The team
then used a 10 GB/s network called PacketNet, which was
provided by National Lambda Rail.
In the past, UDT and technologies could move data at
high speeds but faced challenges when utilized to transport
data from disk to disk over long distances, according to
Szalay, who said that Sector allowed the team to transport
large data sets from disk to disk quite easily.
"During the challenge, we copied the data over in 20
minutes from disks in Chicago to disks in Florida, ready to
be used immediately," he explained. "Previously, we would
have had to ship disks around, which is a cumbersome
process involving a lot of work on either end."
The Johns Hopkins team — Szalay, Ani Thakar, Jan
vandenBerg and Alainna Wonders — and the researchers
at University of Illinois would eventually like to use such
network speeds as part of their everyday research.
Michael McCarty, chief network officer for Johns
Hopkins, said he believes that such collaboration could
lead to benefits for not only Johns Hopkins researchers but
the community at large.
"We have been excited to have an opportunity to work
with Alex and other scientists across Hopkins whose
research creates new challenges for us as well. Today, we
have partnerships with the state of Maryland, several
Maryland universities and the Internet 2 community that we
have been able to leverage to help make efforts such as
this possible," McCarty said.
GO TO