One of the major tasks of data engineers deals with the problem of automatically organizing data in a meaningful way so that datasets become useful for a variety of applications in the “smart world”, artificial intelligence and analytics of all kinds. One key aspect to data profiling are Unique Column Combinations (UCC). They identify entities in datasets and support different data organizing activities. Up until recently, it was only possible to discover UCCs for moderately sized datasets with quite some run time effort. For large datasets UCCs were not discoverable due to runtime and memory limitations. Researchers at HPI have developed a novel UCC discovery algorithm (“HPIValid”), which drastically reduces the runtime of UCC discovery by orders of magnitude and at the same time reduce the memory footprint in comparision to state of the art algorithms. Across different moderately sized datasets HPIValid performed 5-100 times faster by using up only 5-20 % memory on average. Read more…

Experiments at HPI have shown that the application of heuristic algorithms for the optimization of submodular functions, which can be implemented to optimize traffic, use raw materials in production or allocate goods in markets, reduce the runtime in comparison to traditional algorithms by a factor of 288. While traditional software takes two days to compute the solution for this problem, the heuristic algorithm took only ten minutes and therefore reduced the energy consumption of the computing device to a mere 0.35 % of the original setup.

Next-generation data centers are embracing an increasingly diverse landscape of accelerators and hardware architectures, each offering advantages for certain algorithm classes or application domains. Unfortunately, today‘s software widely ignores this degree of heterogeneity. By executing workloads on the best-suited hardware, power efficiency can be improved significantly, e.g. by a factor of 10 for weather simulation models using FPGA accelerators instead of general-purpose processors.