MIT researchers use system of flash storage devices to handle Big Data

As computers enter ever more areas of our daily lives, the amount of data they produce has grown enormously. But for this “big data” to be useful it must first be analyzed, meaning it needs to be stored in such a way that it can be accessed quickly when required. Previously, any data that needed to be accessed in a hurry would be stored in a computer’s main memory, or dynamic random access memory, but the size of the datasets now being produced makes this impossible.

An MIT research team next month will show off a networked system of flash storage devices they say beats relying on DRAM and networked hard disks to handle Big Data demands. The copious amounts of data now collected for analyzation by organizations overtaxes computers’ main memory, but linking hard disks across an Ethernet network to solve the problem proves too slow, according to the researchers. Their Blue Database Machine, or BlueDBM (sounds like an IBM product!), consists of flash devices controlled by serially networked field-programmable gate array chips that can also process data. The researchers say flash systems can find random pieces of information from within large data sets in microseconds, whereas the seek time for hard disks can be more than double that.