By Jason Snell

April 10, 2019 9:00 AM PT

Seeing a black hole with half a ton of hard drives

Note: This story has not been updated for several years.

In a legitimately amazing achievement for humanity, the Event Horizon Telescope Collaboration announced that it has directly imaged a black hole for the first time¹. There is some amazing science going on here, as this really caps more than a century of the expansion of our understanding of the universe, from Einstein’s general relativity to Eddington’s measurements confirming Einstein’s theory to the first detection of a black hole collision via gravitational waves to, today, this image of a black hole four times the size of our solar system nestled at the center of a galaxy 55 million light-years away from us.

But this is Six Colors, so I want to talk about computers!

To capture this image, the EHT used seven different radio telescopes all around the world in order to use something called interferometry, which combines data from telescopes spread out over a wide distance to essentially create a virtual telescope the size of the distance between the telescopes. The result is a telescope that’s basically the size of Earth. (Among the telescopes used is one at the South Pole, which needed to be retrofitted to make these measurements.)

Then the telescopes have to capture data simultaneously, which means the weather needs to be good in Hawaii and Spain and Chile and the South Pole and other places simultaneously. And when that data is captured, it needs to be brought back to a correlation facility to process it and generate a single data set.

Here’s how Dan Marrone, Associate Professor of Astronomy at the University of Arizona, described it during today’s press conference:

At the end of that, we had five petabytes of data recorded… it amounts to more than half a ton of hard drives. Five petabytes is a lot of data. It’s equivalent to 5,000 years of MP3 files, or according to one study I read, the entire selfie collection over a lifetime for 40,000 people.

The image you saw, though, isn’t five petabytes in size, it’s a few hundred kilobytes. So our data analysis has to collapse this five petabytes of data into an image that’s more than a billion times smaller. We do that in many steps. The first of those steps is to get [hard drive modules] to our correlators in western Massachusetts and Bonn, Germany. The fastest way to do that is not over the Internet, it’s to put them on planes. There’s no Internet that can compete with petabytes of data on a plane.

Let’s do some math. Hawaii is 5,000 miles away from the MIT Haystack correlation facility. Let’s assume roughly 700 terabytes of data (one-seventh the total) is flying from Hawaii to Haystack. It’s about 10 hours to fly from Hawaii to Boston (there are no commercial direct flights from Hilo to Boston so you might want to build in another two hours to fly from Hilo to Honolulu and ride the Wiki Wiki bus and wait for a flight), it takes an hour to drive down from the Mauna Kea summit to Hilo, and another hour to drive from Logan Airport to Groton, MA. Let’s generously estimate it takes 14 hours to get 150 pounds of hard drives from the summit of Mauna Kea to MIT Haystack.

That would mean they transferred 700 terabytes in 50,400 seconds, for a final data rate of about 14 gigabytes per second. As Marrone said, if you’re dealing in petabytes of data, the fastest bandwidth you can buy is not the Internet—it’s putting your hard drives on an airplane and flying them to your destination. (The project did need to wait for Antarctic summer before the data from the South Pole could be flown back—a much slower data rate, though still faster than any other transfer method from Antarctica!)

My pal John Siracusa pointed me to this excellent quote from computer scientist Andrew S. Tanenbaum: “Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.”

Want to know more about black holes? Kip Thorne’s book Black Holes & Time Warps is great.

If you appreciate articles like this one, support us by becoming a Six Colors subscriber. Subscribers get access to an exclusive podcast, members-only stories, and a special community.