Technologies for dealing with information overload : an engineers' point of view

In this chapter, we provide an overview of the technological side of the information overload problem. We discuss the challenges and opportunities offered by the ever-growing and emerging stream of information from an engineering point of view. More concretely, we survey storage and querying techniques for semistructured data, data mining, and information retrieval for analyzing large data collections, and then survey stream processing techniques for online handling of continuously flowing data. In this way, we cover a whole spectrum of different levels of "structuredness" of data. At one end of the spectrum, there is data that, although only loosely structured, can still be organized in some way. At the other end, there are data streams that come in at such a fast pace that even storing them is no longer a valid option. Rather, we need to rely on immediate processing and approximate methods in order to be able to distill information from them. In between these two extremes, we have information retrieval and data mining. For all four domains, we survey the main challenges and give some insights into recent developments and techniques. We also show that from an engineering point of view, information overload is not always considered to be a problem, but that it can also offer many opportunities.