2014

Advancing Big Data Benchmarks - Proceedings of the 2013 Workshop Series on Big Data Benchmarking, WBDB.cn, Xi’an, China, July 16-17, 2013 and WBDB.us, San José, CA, USA, October 9-10, 2013 Revised Selected Papers Rabl, Tilmann; Jacobsen, Hans-Arno; Nambiar, Raghunath; Poess, Meikel; Bhandarkar, Milind A.; Baru, Chaitanya K. in Lecture Notes in Computer Science (2014). (Vol. 8585) Springer.
[ BibTeX ] [ EndNote ] [ URL ]
 
@proceedings{DBLP:conf/wbdb/2013, editor = {Rabl, Tilmann and Jacobsen, Hans{-}Arno and Nambiar, Raghunath and Poess, Meikel and Bhandarkar, Milind A. and Baru, Chaitanya K.}, keywords = {BigData}, publisher = {Springer}, series = {Lecture Notes in Computer Science}, title = {Advancing Big Data Benchmarks - Proceedings of the 2013 Workshop Series on Big Data Benchmarking, WBDB.cn, Xi'an, China, July 16-17, 2013 and WBDB.us, San Jos{{é}}, CA, USA, October 9-10, 2013 Revised Selected Papers}, volume = 8585, year = 2014 }
%0 Conference Proceedings %1 DBLP:conf/wbdb/2013 %B Lecture Notes in Computer Science %D 2014 %E Rabl, Tilmann %E Jacobsen, Hans-Arno %E Nambiar, Raghunath %E Poess, Meikel %E Bhandarkar, Milind A. %E Baru, Chaitanya K. %I Springer %R 10.1007/978-3-319-10596-3 %T Advancing Big Data Benchmarks - Proceedings of the 2013 Workshop Series on Big Data Benchmarking, WBDB.cn, Xi'an, China, July 16-17, 2013 and WBDB.us, San José, CA, USA, October 9-10, 2013 Revised Selected Papers %U https://doi.org/10.1007/978-3-319-10596-3 %V 8585 %@ 978-3-319-10595-6

Specifying Big Data Benchmarks - First Workshop, WBDB 2012, San Jose, CA, USA, May 8-9, 2012, and Second Workshop, WBDB 2012, Pune, India, December 17-18, 2012, Revised Selected Papers Rabl, Tilmann; Poess, Meikel; Baru, Chaitanya K.; Jacobsen, Hans-Arno in Lecture Notes in Computer Science (2014). (Vol. 8163) Springer.
[ BibTeX ] [ EndNote ] [ URL ]
 
@proceedings{DBLP:conf/wbdb/2012, editor = {Rabl, Tilmann and Poess, Meikel and Baru, Chaitanya K. and Jacobsen, Hans{-}Arno}, keywords = {BigData}, publisher = {Springer}, series = {Lecture Notes in Computer Science}, title = {Specifying Big Data Benchmarks - First Workshop, {WBDB} 2012, San Jose, CA, USA, May 8-9, 2012, and Second Workshop, {WBDB} 2012, Pune, India, December 17-18, 2012, Revised Selected Papers}, volume = 8163, year = 2014 }
%0 Conference Proceedings %1 DBLP:conf/wbdb/2012 %B Lecture Notes in Computer Science %D 2014 %E Rabl, Tilmann %E Poess, Meikel %E Baru, Chaitanya K. %E Jacobsen, Hans-Arno %I Springer %R 10.1007/978-3-642-53974-9 %T Specifying Big Data Benchmarks - First Workshop, WBDB 2012, San Jose, CA, USA, May 8-9, 2012, and Second Workshop, WBDB 2012, Pune, India, December 17-18, 2012, Revised Selected Papers %U https://doi.org/10.1007/978-3-642-53974-9 %V 8163 %@ 978-3-642-53973-2

Towards a Complete BigBench Implementation. Rabl, Tilmann; Frank, Michael; Danisch, Manuel; Gowda, Bhaskar; Jacobsen, Hans-Arno (2014). 3–11.
[ Abstract ] [ BibTeX ] [ EndNote ] [ Download ]
 
BigBench was the first proposal for an end-to-end big data analytics benchmark. It features a set of 30 realistic queries based on real big data use cases. It was fully specified and completely implemented on the Hadoop stack. In this paper, we present updates on our development of a complete implementation on the Hadoop ecosystem. We will focus on the changes that we have made to data set, scaling, refresh process, and metric.
@inproceedings{DBLP:conf/wbdb/RablFDGJ14, abstract = {BigBench was the first proposal for an end-to-end big data analytics benchmark. It features a set of 30 realistic queries based on real big data use cases. It was fully specified and completely implemented on the Hadoop stack. In this paper, we present updates on our development of a complete implementation on the Hadoop ecosystem. We will focus on the changes that we have made to data set, scaling, refresh process, and metric.}, author = {Rabl, Tilmann and Frank, Michael and Danisch, Manuel and Gowda, Bhaskar and Jacobsen, Hans{-}Arno}, booktitle = {Big Data Benchmarking - 5th International Workshop, {WBDB} 2014, Potsdam, Germany, August 5-6, 2014, Revised Selected Papers}, crossref = {DBLP:conf/wbdb/2014}, keywords = {WBDB bigbench}, pages = {3-11}, title = {Towards a Complete BigBench Implementation}, year = 2014 }
%0 Conference Paper %1 DBLP:conf/wbdb/RablFDGJ14 %A Rabl, Tilmann %A Frank, Michael %A Danisch, Manuel %A Gowda, Bhaskar %A Jacobsen, Hans-Arno %B Big Data Benchmarking - 5th International Workshop, WBDB 2014, Potsdam, Germany, August 5-6, 2014, Revised Selected Papers %D 2014 %P 3-11 %T Towards a Complete BigBench Implementation %X BigBench was the first proposal for an end-to-end big data analytics benchmark. It features a set of 30 realistic queries based on real big data use cases. It was fully specified and completely implemented on the Hadoop stack. In this paper, we present updates on our development of a complete implementation on the Hadoop ecosystem. We will focus on the changes that we have made to data set, scaling, refresh process, and metric.

Discussion of BigBench: A Proposed Industry Standard Performance Benchmark for Big Data. Baru, Chaitanya K.; Bhandarkar, Milind A.; Curino, Carlo; Danisch, Manuel; Frank, Michael; Gowda, Bhaskar; Jacobsen, Hans-Arno; Jie, Huang; Kumar, Dileep; Nambiar, Raghunath Othayoth; Poess, Meikel; Raab, Francois; Rabl, Tilmann; Ravi, Nishkam; Sachs, Kai; Sen, Saptak; Yi, Lan; Youn, Choonhan (2014). 44–63.
[ BibTeX ] [ EndNote ] [ URL ]
 
@inproceedings{DBLP:conf/tpctc/BaruBCDFGJJKNPRRRSSYY14, author = {Baru, Chaitanya K. and Bhandarkar, Milind A. and Curino, Carlo and Danisch, Manuel and Frank, Michael and Gowda, Bhaskar and Jacobsen, Hans{-}Arno and Jie, Huang and Kumar, Dileep and Nambiar, Raghunath Othayoth and Poess, Meikel and Raab, Francois and Rabl, Tilmann and Ravi, Nishkam and Sachs, Kai and Sen, Saptak and Yi, Lan and Youn, Choonhan}, booktitle = {Performance Characterization and Benchmarking. Traditional to Big Data - 6th {TPC} Technology Conference, {TPCTC} 2014, Hangzhou, China, September 1-5, 2014. Revised Selected Papers}, crossref = {DBLP:conf/tpctc/2014}, keywords = {TPCTC}, pages = {44--63}, title = {Discussion of BigBench: {A} Proposed Industry Standard Performance Benchmark for Big Data}, year = 2014 }
%0 Conference Paper %1 DBLP:conf/tpctc/BaruBCDFGJJKNPRRRSSYY14 %A Baru, Chaitanya K. %A Bhandarkar, Milind A. %A Curino, Carlo %A Danisch, Manuel %A Frank, Michael %A Gowda, Bhaskar %A Jacobsen, Hans-Arno %A Jie, Huang %A Kumar, Dileep %A Nambiar, Raghunath Othayoth %A Poess, Meikel %A Raab, Francois %A Rabl, Tilmann %A Ravi, Nishkam %A Sachs, Kai %A Sen, Saptak %A Yi, Lan %A Youn, Choonhan %B Performance Characterization and Benchmarking. Traditional to Big Data - 6th TPC Technology Conference, TPCTC 2014, Hangzhou, China, September 1-5, 2014. Revised Selected Papers %D 2014 %P 44--63 %R 10.1007/978-3-319-15350-6\_4 %T Discussion of BigBench: A Proposed Industry Standard Performance Benchmark for Big Data %U https://doi.org/10.1007/978-3-319-15350-6_4

PSBench: A Benchmark for Content- and Topic-Based Publish/Subscribe Systems. Zhang, Kaiwen; Rabl, Tilmann; Sun, Yi Ping; Kumar, Rushab; Zen, Nayeem; Jacobsen, Hans-Arno (2014). 17–18.
[ BibTeX ] [ EndNote ] [ URL ]
 
@inproceedings{DBLP:conf/middleware/ZhangRSKZJ14, author = {Zhang, Kaiwen and Rabl, Tilmann and Sun, Yi Ping and Kumar, Rushab and Zen, Nayeem and Jacobsen, Hans{-}Arno}, booktitle = {Proceedings of the Middleware '14 Posters {\&} Demos Session, Bordeaux, France, December 8-12, 2014}, crossref = {DBLP:conf/middleware/2014pd}, keywords = {Middleware}, pages = {17-18}, title = {PSBench: A Benchmark for Content- and Topic-Based Publish/Subscribe Systems}, year = 2014 }
%0 Conference Paper %1 DBLP:conf/middleware/ZhangRSKZJ14 %A Zhang, Kaiwen %A Rabl, Tilmann %A Sun, Yi Ping %A Kumar, Rushab %A Zen, Nayeem %A Jacobsen, Hans-Arno %B Proceedings of the Middleware '14 Posters & Demos Session, Bordeaux, France, December 8-12, 2014 %D 2014 %P 17-18 %R 10.1145/2678508.2678517 %T PSBench: A Benchmark for Content- and Topic-Based Publish/Subscribe Systems %U https://doi.org/10.1145/2678508.2678517

CaSSanDra: An SSD Boosted Key-Value Store. Menon, Prashanth; Rabl, Tilmann; Sadoghi, Mohammad; Jacobsen, Hans-Arno (2014). 1162–1167.
[ Abstract ] [ BibTeX ] [ EndNote ] [ Download ]
 
With the ever growing size and complexity of enterprise systems there is a pressing need for more detailed application performance management. Due to the high data rates, traditional database technology cannot sustain the required performance. Alternatives are the more lightweight and, thus, more performant key-value stores. However, these systems tend to sacrifice read performance in order to obtain the desired write throughput by avoiding random disk access in favor of fast sequential accesses. With the advent of SSDs, built upon the philosophy of no moving parts, the boundary between sequential vs. random access is now becoming blurred. This provides a unique opportunity to extend the storage memory hierarchy using SSDs in key-value stores. In this paper, we extensively evaluate the benefits of using SSDs in commercialized key-value stores. In particular, we investigate the performance of hybrid SSD-HDD systems and demonstrate the benefits of our SSD caching and our novel dynamic schema model.
@inproceedings{DBLP:conf/icde/MenonRSJ14, abstract = {With the ever growing size and complexity of enterprise systems there is a pressing need for more detailed application performance management. Due to the high data rates, traditional database technology cannot sustain the required performance. Alternatives are the more lightweight and, thus, more performant key-value stores. However, these systems tend to sacrifice read performance in order to obtain the desired write throughput by avoiding random disk access in favor of fast sequential accesses. With the advent of SSDs, built upon the philosophy of no moving parts, the boundary between sequential vs. random access is now becoming blurred. This provides a unique opportunity to extend the storage memory hierarchy using SSDs in key-value stores. In this paper, we extensively evaluate the benefits of using SSDs in commercialized key-value stores. In particular, we investigate the performance of hybrid SSD-HDD systems and demonstrate the benefits of our SSD caching and our novel dynamic schema model.}, author = {Menon, Prashanth and Rabl, Tilmann and Sadoghi, Mohammad and Jacobsen, Hans{-}Arno}, booktitle = {{IEEE} 30th International Conference on Data Engineering, Chicago, {ICDE} 2014, IL, USA, March 31 - April 4, 2014}, crossref = {DBLP:conf/icde/2014}, keywords = {ICDE}, pages = {1162-1167}, title = {CaSSanDra: An {SSD} Boosted Key-Value Store}, year = 2014 }
%0 Conference Paper %1 DBLP:conf/icde/MenonRSJ14 %A Menon, Prashanth %A Rabl, Tilmann %A Sadoghi, Mohammad %A Jacobsen, Hans-Arno %B IEEE 30th International Conference on Data Engineering, Chicago, ICDE 2014, IL, USA, March 31 - April 4, 2014 %D 2014 %P 1162-1167 %T CaSSanDra: An SSD Boosted Key-Value Store %X With the ever growing size and complexity of enterprise systems there is a pressing need for more detailed application performance management. Due to the high data rates, traditional database technology cannot sustain the required performance. Alternatives are the more lightweight and, thus, more performant key-value stores. However, these systems tend to sacrifice read performance in order to obtain the desired write throughput by avoiding random disk access in favor of fast sequential accesses. With the advent of SSDs, built upon the philosophy of no moving parts, the boundary between sequential vs. random access is now becoming blurred. This provides a unique opportunity to extend the storage memory hierarchy using SSDs in key-value stores. In this paper, we extensively evaluate the benefits of using SSDs in commercialized key-value stores. In particular, we investigate the performance of hybrid SSD-HDD systems and demonstrate the benefits of our SSD caching and our novel dynamic schema model.

Optimizing Key-Value Stores for Hybrid Storage Architectures. Menon, Prashanth; Rabl, Tilmann; Sadoghi, Mohammad; Jacobsen, Hans-Arno (2014). 355–358.
[ BibTeX ] [ EndNote ] [ URL ]
 
@inproceedings{DBLP:conf/cascon/MenonRSJ14, author = {Menon, Prashanth and Rabl, Tilmann and Sadoghi, Mohammad and Jacobsen, Hans{-}Arno}, booktitle = {Proceedings of 24th Annual International Conference on Computer Science and Software Engineering, {CASCON} 2014, Markham, Ontario, Canada, 3-5 November, 2014}, crossref = {DBLP:conf/cascon/2014}, keywords = {CASCON}, pages = {355-358}, title = {Optimizing Key-Value Stores for Hybrid Storage Architectures}, year = 2014 }
%0 Conference Paper %1 DBLP:conf/cascon/MenonRSJ14 %A Menon, Prashanth %A Rabl, Tilmann %A Sadoghi, Mohammad %A Jacobsen, Hans-Arno %B Proceedings of 24th Annual International Conference on Computer Science and Software Engineering, CASCON 2014, Markham, Ontario, Canada, 3-5 November, 2014 %D 2014 %P 355-358 %T Optimizing Key-Value Stores for Hybrid Storage Architectures %U http://dl.acm.org/citation.cfm?id=2735582

Materialized Views in Cassandra. Rabl, Tilmann; Jacobsen, Hans-Arno (2014). 351–354.
[ Abstract ] [ BibTeX ] [ EndNote ] [ Download ]
 
Many web companies deal with enormous data sizes and request rates beyond the capabilities of traditional database systems. This has led to the development of modern Big Data Platforms (BDPs). BDPs handle large amounts of data and activity through massively distributed infrastructures. To achieve performance and availability at Internet scale, BDPs restrict querying capability, and provide weaker consistency guarantees than traditional ACID transactions. The reduced functionality as found in key-value stores is sufficient for many web applications. An important requirement of many big data systems is an online view of the current status of the data and activity. Typical big data systems such as key-value stores only allow a key-based access. In order to enable more complex querying mechanisms, while satisfying necessary latencies materialized views are employed. The efficiency of the maintenance of these views is a key factor of the usability of the system. Expensive operations such as full table scans are impractical for small, frequent modifications on Internet-scale data sets. In this paper, we present an efficient implementation of materialized views in key-value stores that enables complex query processing and is tailored for efficient maintenance.
@inproceedings{DBLP:conf/cascon/RablJ14, abstract = {Many web companies deal with enormous data sizes and request rates beyond the capabilities of traditional database systems. This has led to the development of modern Big Data Platforms (BDPs). BDPs handle large amounts of data and activity through massively distributed infrastructures. To achieve performance and availability at Internet scale, BDPs restrict querying capability, and provide weaker consistency guarantees than traditional ACID transactions. The reduced functionality as found in key-value stores is sufficient for many web applications. An important requirement of many big data systems is an online view of the current status of the data and activity. Typical big data systems such as key-value stores only allow a key-based access. In order to enable more complex querying mechanisms, while satisfying necessary latencies materialized views are employed. The efficiency of the maintenance of these views is a key factor of the usability of the system. Expensive operations such as full table scans are impractical for small, frequent modifications on Internet-scale data sets. In this paper, we present an efficient implementation of materialized views in key-value stores that enables complex query processing and is tailored for efficient maintenance.}, author = {Rabl, Tilmann and Jacobsen, Hans{-}Arno}, booktitle = {Proceedings of 24th Annual International Conference on Computer Science and Software Engineering, {CASCON} 2014, Markham, Ontario, Canada, 3-5 November, 2014}, crossref = {DBLP:conf/cascon/2014}, keywords = {CASCON}, pages = {351-354}, title = {Materialized Views in Cassandra}, year = 2014 }
%0 Conference Paper %1 DBLP:conf/cascon/RablJ14 %A Rabl, Tilmann %A Jacobsen, Hans-Arno %B Proceedings of 24th Annual International Conference on Computer Science and Software Engineering, CASCON 2014, Markham, Ontario, Canada, 3-5 November, 2014 %D 2014 %P 351-354 %T Materialized Views in Cassandra %X Many web companies deal with enormous data sizes and request rates beyond the capabilities of traditional database systems. This has led to the development of modern Big Data Platforms (BDPs). BDPs handle large amounts of data and activity through massively distributed infrastructures. To achieve performance and availability at Internet scale, BDPs restrict querying capability, and provide weaker consistency guarantees than traditional ACID transactions. The reduced functionality as found in key-value stores is sufficient for many web applications. An important requirement of many big data systems is an online view of the current status of the data and activity. Typical big data systems such as key-value stores only allow a key-based access. In order to enable more complex querying mechanisms, while satisfying necessary latencies materialized views are employed. The efficiency of the maintenance of these views is a key factor of the usability of the system. Expensive operations such as full table scans are impractical for small, frequent modifications on Internet-scale data sets. In this paper, we present an efficient implementation of materialized views in key-value stores that enables complex query processing and is tailored for efficient maintenance.

DGFIndex for Smart Grid: Enhancing Hive with a Cost-Effective Multidimensional Range Index. Liu, Yue; Hu, Songlin; Rabl, Tilmann; Liu, Wantao; Jacobsen, Hans-Arno; Wu, Kaifeng; Chen, Jian; Li, Jintao in PVLDB (2014). 7(13) 1496–1507.
[ Abstract ] [ BibTeX ] [ EndNote ] [ Download ]
 
In Smart Grid applications, as the number of deployed electric smart meters increases, massive amounts of valuable meter data is generated and collected every day. To enable reliable data collection and make business decisions fast, high throughput storage and high-performance analysis of massive meter data become crucial for grid companies. Considering the advantage of high efficiency, fault tolerance, and price-performance of Hadoop and Hive systems, they are frequently deployed as underlying platform for big data processing. However, in real business use cases, these data analysis applications typically involve multidimensional range queries (MDRQ) as well as batch reading and statistics on the meter data. While Hive is high-performance at complex data batch reading and analysis, it lacks efficient indexing techniques for MDRQ. In this paper, we propose DGFIndex, an index structure for Hive that efficiently supports MDRQ for massive meter data. DGFIndex divides the data space into cubes using the grid file technique. Unlike the existing indexes in Hive, which stores all combinations of multiple dimensions, DGFIndex only stores the information of cubes. This leads to smaller index size and faster query processing. Furthermore, with pre-computing user-defined aggregations of each cube, DGFIndex only needs to access the boundary region for aggregation query. Our comprehensive experiments show that DGFIndex can save significant disk space in comparison with the existing indexes in Hive and the query performance with DGFIndex is 2-63 times faster than existing indexes in Hive, 2-94 times faster than HadoopDB, 2-75 times faster than scanning the whole table in different query selectivity.
@article{DBLP:journals/pvldb/LiuHRLJWCL14, abstract = {In Smart Grid applications, as the number of deployed electric smart meters increases, massive amounts of valuable meter data is generated and collected every day. To enable reliable data collection and make business decisions fast, high throughput storage and high-performance analysis of massive meter data become crucial for grid companies. Considering the advantage of high efficiency, fault tolerance, and price-performance of Hadoop and Hive systems, they are frequently deployed as underlying platform for big data processing. However, in real business use cases, these data analysis applications typically involve multidimensional range queries (MDRQ) as well as batch reading and statistics on the meter data. While Hive is high-performance at complex data batch reading and analysis, it lacks efficient indexing techniques for MDRQ. In this paper, we propose DGFIndex, an index structure for Hive that efficiently supports MDRQ for massive meter data. DGFIndex divides the data space into cubes using the grid file technique. Unlike the existing indexes in Hive, which stores all combinations of multiple dimensions, DGFIndex only stores the information of cubes. This leads to smaller index size and faster query processing. Furthermore, with pre-computing user-defined aggregations of each cube, DGFIndex only needs to access the boundary region for aggregation query. Our comprehensive experiments show that DGFIndex can save significant disk space in comparison with the existing indexes in Hive and the query performance with DGFIndex is 2-63 times faster than existing indexes in Hive, 2-94 times faster than HadoopDB, 2-75 times faster than scanning the whole table in different query selectivity.}, author = {Liu, Yue and Hu, Songlin and Rabl, Tilmann and Liu, Wantao and Jacobsen, Hans{-}Arno and Wu, Kaifeng and Chen, Jian and Li, Jintao}, journal = {{PVLDB}}, keywords = {PVLDB}, number = 13, pages = {1496-1507}, title = {DGFIndex for Smart Grid: Enhancing Hive with a Cost-Effective Multidimensional Range Index}, volume = 7, year = 2014 }
%0 Journal Article %1 DBLP:journals/pvldb/LiuHRLJWCL14 %A Liu, Yue %A Hu, Songlin %A Rabl, Tilmann %A Liu, Wantao %A Jacobsen, Hans-Arno %A Wu, Kaifeng %A Chen, Jian %A Li, Jintao %D 2014 %J PVLDB %N 13 %P 1496-1507 %T DGFIndex for Smart Grid: Enhancing Hive with a Cost-Effective Multidimensional Range Index %V 7 %X In Smart Grid applications, as the number of deployed electric smart meters increases, massive amounts of valuable meter data is generated and collected every day. To enable reliable data collection and make business decisions fast, high throughput storage and high-performance analysis of massive meter data become crucial for grid companies. Considering the advantage of high efficiency, fault tolerance, and price-performance of Hadoop and Hive systems, they are frequently deployed as underlying platform for big data processing. However, in real business use cases, these data analysis applications typically involve multidimensional range queries (MDRQ) as well as batch reading and statistics on the meter data. While Hive is high-performance at complex data batch reading and analysis, it lacks efficient indexing techniques for MDRQ. In this paper, we propose DGFIndex, an index structure for Hive that efficiently supports MDRQ for massive meter data. DGFIndex divides the data space into cubes using the grid file technique. Unlike the existing indexes in Hive, which stores all combinations of multiple dimensions, DGFIndex only stores the information of cubes. This leads to smaller index size and faster query processing. Furthermore, with pre-computing user-defined aggregations of each cube, DGFIndex only needs to access the boundary region for aggregation query. Our comprehensive experiments show that DGFIndex can save significant disk space in comparison with the existing indexes in Hive and the query performance with DGFIndex is 2-63 times faster than existing indexes in Hive, 2-94 times faster than HadoopDB, 2-75 times faster than scanning the whole table in different query selectivity.

TPC-DI: The First Industry Benchmark for Data Integration. Poess, Meikel; Rabl, Tilmann; Jacobsen, Hans-Arno; Caufield, Brian in PVLDB (2014). 7(13) 1367–1378.
[ Abstract ] [ BibTeX ] [ EndNote ] [ Download ]
 
Historically, the process of synchronizing a decision support system with data from operational systems has been referred to as Extract, Transform, Load (ETL) and the tools supporting such process have been referred to as ETL tools. Recently, ETL was replaced by the more comprehensive acronym, data integration (DI). DI describes the process of extracting and combining data from a variety of data source formats, transforming that data into a unified data model representation and loading it into a data store. This is done in the context of a variety of scenarios, such as data acquisition for business intelligence, analytics and data warehousing, but also synchronization of data between operational applications, data migrations and conversions, master data management, enterprise data sharing and delivery of data services in a service-oriented architecture context, amongst others. With these scenarios relying on up-to-date information it is critical to implement a highly performing, scalable and easy to maintain data integration system. This is especially important as the complexity, variety and volume of data is constantly increasing and performance of data integration systems is becoming very critical. Despite the significance of having a highly performing DI system, there has been no industry standard for measuring and comparing their performance. The TPC, acknowledging this void, has released TPC-DI, an innovative benchmark for data integration. This paper motivates the reasons behind its development, describes its main characteristics including workload, run rules, metric, and explains key decisions.
@article{DBLP:journals/pvldb/PoessRC14, abstract = {Historically, the process of synchronizing a decision support system with data from operational systems has been referred to as Extract, Transform, Load (ETL) and the tools supporting such process have been referred to as ETL tools. Recently, ETL was replaced by the more comprehensive acronym, data integration (DI). DI describes the process of extracting and combining data from a variety of data source formats, transforming that data into a unified data model representation and loading it into a data store. This is done in the context of a variety of scenarios, such as data acquisition for business intelligence, analytics and data warehousing, but also synchronization of data between operational applications, data migrations and conversions, master data management, enterprise data sharing and delivery of data services in a service-oriented architecture context, amongst others. With these scenarios relying on up-to-date information it is critical to implement a highly performing, scalable and easy to maintain data integration system. This is especially important as the complexity, variety and volume of data is constantly increasing and performance of data integration systems is becoming very critical. Despite the significance of having a highly performing DI system, there has been no industry standard for measuring and comparing their performance. The TPC, acknowledging this void, has released TPC-DI, an innovative benchmark for data integration. This paper motivates the reasons behind its development, describes its main characteristics including workload, run rules, metric, and explains key decisions.}, author = {Poess, Meikel and Rabl, Tilmann and Jacobsen, Hans{-}Arno and Caufield, Brian}, journal = {{PVLDB}}, keywords = {PVLDB}, number = 13, pages = {1367-1378}, title = {{TPC-DI:} The First Industry Benchmark for Data Integration}, volume = 7, year = 2014 }
%0 Journal Article %1 DBLP:journals/pvldb/PoessRC14 %A Poess, Meikel %A Rabl, Tilmann %A Jacobsen, Hans-Arno %A Caufield, Brian %D 2014 %J PVLDB %N 13 %P 1367-1378 %T TPC-DI: The First Industry Benchmark for Data Integration %V 7 %X Historically, the process of synchronizing a decision support system with data from operational systems has been referred to as Extract, Transform, Load (ETL) and the tools supporting such process have been referred to as ETL tools. Recently, ETL was replaced by the more comprehensive acronym, data integration (DI). DI describes the process of extracting and combining data from a variety of data source formats, transforming that data into a unified data model representation and loading it into a data store. This is done in the context of a variety of scenarios, such as data acquisition for business intelligence, analytics and data warehousing, but also synchronization of data between operational applications, data migrations and conversions, master data management, enterprise data sharing and delivery of data services in a service-oriented architecture context, amongst others. With these scenarios relying on up-to-date information it is critical to implement a highly performing, scalable and easy to maintain data integration system. This is especially important as the complexity, variety and volume of data is constantly increasing and performance of data integration systems is becoming very critical. Despite the significance of having a highly performing DI system, there has been no industry standard for measuring and comparing their performance. The TPC, acknowledging this void, has released TPC-DI, an innovative benchmark for data integration. This paper motivates the reasons behind its development, describes its main characteristics including workload, run rules, metric, and explains key decisions.