News: Thank you for visiting HyPer's research site hyper-db.de. The commercial license for HyPer, a spin-off of TUM, has been acquired by Tableau Software, a global leader in rapid-fire, easy-to-use business analytics software. For more information, please visit the Tableau web site.

Highlights

In-memory Data Management

HyPer relies on in-memory data management without the ballast of traditional database systems caused by DBMS-controlled page structures and buffer management. SQL table definitions are transformed into simple vector-based virtual memory representations – which constitutes a column oriented physical storage scheme.

Data-centric Code Generation

Transactions and queries are specified in SQL or a PL/SQL-like scripting language and are efficiently compiled into efficient LLVM assembly code.

Multi-Version Concurrency Control

OLAP query processing is separated from mission-critical OLTP transaction processing using multi-version concurrency control (MVCC).

No compromises

HyPer's transaction processing is fully ACID-compliant. Queries are specified in SQL-92 plus some extensions from subsequent standards.


Team

Head: Prof. Alfons Kemper, Prof. Thomas Neumann

Ph.D. Students: Jan Finis, Moritz Kaufmann, Andreas Kipf, Timo Kersten, Harald Lang, Viktor Leis, Tobias Mühlbauer, Varun Pandey, Linnea Passing, Wolf Rödiger, Manuel Then, Alexander van Renen

Alumni: Dr. Martina Albutiu, Dr. Veneta Dobreva, Dr. Florian Funke, Dr. Andrey Gubichev, Dr. Henrik Mühe, Dr. Michael Seibold


Publications

Foundational HyPer publications

Venue Publication Link
SIGMOD 2015 Fast Serializable Multi-Version Concurrency Control for Main-Memory Database Systems
Thomas Neumann, Tobias Mühlbauer, Alfons Kemper, 2015.
pdf
VLDB 2011 Efficiently Compiling Efficient Query Plans for Modern Hardware
Thomas Neumann, 2011.
pdf

List of all publications

Venue Publication Link
SIGMOD 2016 High-Performance Geospatial Analytics in HyPerSpace (Demo)
Varun Pandey, Andreas Kipf, Dimitri Vorona, Tobias Mühlbauer, Thomas Neumann, Alfons Kemper, 2016.
preprint
ICDE 2016 Flow-Join: Adaptive Skew Handling for Distributed Joins over High-Speed Networks
Wolf Rödiger, Sam Idicula, Alfons Kemper, Thomas Neumann, 2016.
pdf
SIGMOD 2016 Data Blocks: Hybrid OLTP and OLAP on Compressed Storage using both Vectorization and Compilation
Harald Lang, Tobias Mühlbauer, Florian Funke, Peter Boncz, Thomas Neumann, Alfons Kemper, 2016.
preprint
VLDB 2016 High-Speed Query Processing over High-Speed Networks
Wolf Rödiger, Tobias Mühlbauer, Alfons Kemper, Thomas Neumann, 2016.
pdf
VLDB 2016 How Good Are Query Optimizers, Really?
Viktor Leis, Andrey Gubichev, Atanas Mirchev, Peter Boncz, Alfons Kemper, Thomas Neumann, 2016.
pdf
FGDB 2015 Efficient Integration of Data and Graph Mining Algorithms in Relational Database Systems
Manuel Then, Linnea Passing, Nina Hubig, Stephan Günnemann, Alfons Kemper, Thomas Neumann, 2015.
VLDB 2015 Efficient Processing of Window Functions in Analytical SQL Queries
Viktor Leis, Kan Kundhikanjana, Alfons Kemper, Thomas Neumann, 2015.
pdf
SIGMOD 2015 Fast Serializable Multi-Version Concurrency Control for Main-Memory Database Systems
Thomas Neumann, Tobias Mühlbauer, Alfons Kemper, 2015.
pdf
DanaC 2015 High-Performance Main-Memory Database Systems and Modern Virtualization: Friends or Foes?
Tobias Mühlbauer, Wolf Rödiger, Andreas Kipf, Thomas Neumann, Alfons Kemper, 2015.
pdf
TKDE Scaling HTM-Supported Database Transactions to Many Cores
Viktor Leis, Alfons Kemper, Thomas Neumann, 2015.
VLDB 2015 The More the Merrier: Efficient Multi-Source Graph Traversal
Manuel Then, Moritz Kaufmann, Fernando Chirigati, Tuan-Anh Hoang-Vu, Kien Pham, Alfons Kemper, Thomas Neumann, Huy T. Vo, 2015.
pdf
BTW 2015 Unnesting Arbitrary Queries
Thomas Neumann, Alfons Kemper, 2015.
pdf
BTW 2015 Hochperformante Analysen in Graph-Datenbanken
Moritz Kaufmann, Tobias Mühlbauer, Manuel Then, Andrey Gubichev, Alfons Kemper, Thomas Neumann, 2015.
pdf
Datenbank Spektrum HyPer Beyond Software: Exploiting Modern Hardware for Main-Memory Database Systems
Florian Funke, Alfons Kemper, Tobias Mühlbauer, Thomas Neumann, Viktor Leis, 2014.
Springer DL
VLDB 2014 Engineering High-Performance Database Engines
Thomas Neumann, 2014.
pdf
DaMoN 2014 Heterogeneity-Conscious Parallel Query Execution: Getting a better mileage while driving faster!
Tobias Mühlbauer, Wolf Rödiger, Robert Seilbeck, Alfons Kemper, Thomas Neumann, 2014.
pdf
SIGMOD 2014 Morsel-Driven Parallelism: A NUMA-Aware Query Evaluation Framework for the Many-Core Age
Viktor Leis, Peter Boncz, Alfons Kemper, Thomas Neumann, 2014.
pdf
SIGMOD 2014 One DBMS for all: the Brawny Few and the Wimpy Crowd (Demonstration)
Tobias Mühlbauer, Wolf Rödiger, Robert Seilbeck, Angelika Reiser, Alfons Kemper, Thomas Neumann, 2014.
pdf
DEBULL Compiling Database Queries into Machine Code
Thomas Neumann, Viktor Leis, Data Engineering Bulletin, March 2014.
pdf
ICDE 2014 Locality-Sensitive Operators for Parallel Main-Memory Database Clusters
Wolf Rödiger, Tobias Mühlbauer, Philipp Unterbrunner, Angelika Reiser, Alfons Kemper, Thomas Neumann, 2014.
pdf
ICDE 2014 Exploiting Hardware Transactional Memory in Main-Memory Databases
Viktor Leis, Alfons Kemper, Thomas Neumann, 2014. Best Paper Award
pdf
VLDB 2014 Instant Loading for Main Memory Databases
Tobias Mühlbauer, Wolf Rödiger, Robert Seilbeck, Angelika Reiser, Alfons Kemper, Thomas Neumann, 2013.
pdf
IMDM 2013 An Evaluation of Strict Timestamp Ordering Concurrency Control for Main-Memory Database Systems
Stephan Wolf, Henrik Mühe, Alfons Kemper, Thomas Neumann, 2013.
IMDM 2013 Massively Parallel NUMA-aware Hash Joins
Harald Lang, Viktor Leis, Martina-Cezara Albutiu, Thomas Neumann, Alfons Kemper, 2013.
DEBULL Transaction Processing in the Hybrid OLTP&OLAP Main-Memory Database System HyPer
Alfons Kemper, Thomas Neumann, Jan Finis, Florian Funke, Viktor Leis, Henrik Mühe, Tobias Mühlbauer, Wolf Rödiger, IEEE Computer Society Data Engineering Bulletin, Special Issue on "Main Memory Databases", 2013.
Issue
DanaC 2013 ScyPer: Elastic OLAP Throughput on Transactional Data
Tobias Mühlbauer, Wolf Rödiger, Angelika Reiser, Alfons Kemper, Thomas Neumann, 2013.
ACM DL
BTW 2013 Extending the MPSM Join
Martina-Cezara Albutiu, Alfons Kemper, Thomas Neumann, 2013.
pdf
BTW 2013 ScyPer: A Hybrid OLTP&OLAP Distributed Main Memory Database System for Scalable Real-Time Analytics (Demonstration)
Tobias Mühlbauer, Wolf Rödiger, Angelika Reiser, Alfons Kemper, Thomas Neumann, 2013.
pdf
CIDR 2013 Executing Long-Running Transactions in Synchronization-Free Main Memory Database Systems
Henrik Mühe and Alfons Kemper and Thomas Neumann, 2013.
pdf
ICDE 2013 CPU and Cache Efficient Management of Memory-Resident Databases
Holger Pirk, Florian Funke, Martin Grund, Thomas Neumann, Ulf Leser, Stefan Manegold, Alfons Kemper, Martin Kersten, 2013.
ICDE 2013 The Adaptive Radix Tree: ARTful Indexing for Main-Memory Databases
Viktor Leis, Alfons Kemper and Thomas Neumann, 2013.
pdf
VLDB 2012 Massively Parallel Sort-Merge Joins in Main Memory Multi-Core Database Systems
Martina-Cezara Albutiu, Alfons Kemper and Thomas Neumann, 2012.
pdf
VLDB 2012 Compacting Transactional Data in Hybrid OLTP&OLAP Databases
Florian Funke, Alfons Kemper, Thomas Neumann, 2012.
pdf
DEBULL HyPer: Adapting Columnar Main -Memory Data Management for Transactional AND Query Processing
Alfons Kemper, Thomas Neumann, Florian Funke, Viktor Leis, Henrik Mühe, Bulletin of the Technical Committee on Data Engineering, March 2012, Vol. 35, No. 1, pp. 46–51.
Issue
Technical Report Massively Parallel Sort-Merge Joins in Main Memory Multi-Core Database Systems
Martina-Cezara Albutiu, Alfons Kemper and Thomas Neumann, Technical Report, TUM-I121, March, 16, 2012.
pdf, pptx
EDBT 2012 The Mainframe Strikes Back: Elastic Multi-Tenancy Using Main Memory Database Systems On A Many-Core Server
Henrik Mühe, Alfons Kemper and Thomas Neumann, 2012.
VLDB 2011 HyPer-sonic Combined Transaction AND Query Processing
Florian Funke and Alfons Kemper and Thomas Neumann, 2011.
VLDB 2011 Efficiently Compiling Efficient Query Plans for Modern Hardware
Thomas Neumann, 2011.
pdf
DBTest 2011 The mixed workload CH-benCHmark
Dagstuhl "Robust Query Processing" Breakout Group "Workload Management", 2011.
DaMoN 2011 How to Efficiently Snapshot Transactional Data: Hardware or Software Controlled?
Henrik Mühe and Alfons Kemper and Thomas Neumann, 2011.
Datenbank Spektrum HyPer: Die effiziente Reinkarnation des Schattenspeichers in einem Hauptspeicher-DBMS
Florian Funke and Alfons Kemper and Henrik Mühe and Thomas Neumann, 2011.
Datenbank Spektrum
BTW 2011 Benchmarking Hybrid OLTP&OLAP Database Systems
Florian Funke and Alfons Kemper and Thomas Neumann, 2011.
ICDE 2011 HyPer: A Hybrid OLTP&OLAP Main Memory Database System Based on Virtual Memory Snapshots
Alfons Kemper and Thomas Neumann, 2011.
IEEE Xplore
Technical Report HyPer - Hybrid OLTP&OLAP High Performance Database System
Alfons Kemper and Thomas Neumann, Technical Report, TUM-I1010, May, 19, 2010.
pdf
SIGMOD 2009 Query simplification: graceful degradation for join-order optimization
Thomas Neumann, 2009.
SIGMOD 2008 Dynamic programming strikes back
Guido Moerkotte and Thomas Neumann, 2008.
VLDB 2006 Analysis of Two Existing and One New Dynamic Programming Algorithm for the Generation of Optimal Bushy Join Trees without Cross Products.
Guido Moerkotte and Thomas Neumann, 2006.

Presentations/Mentions

Date Venue
May 21, 2010 Colloquium of the Chair of Database Systems
May 26, 2010 "Grundlagen von Datenbanken " Workshop (GvDB, Bad Helmstedt)
June 26, 2010 IBM Böblingen
July 22, 2010 Inaugural Lecture ("Antrittsvorlesung" Thomas Neumann)
August 13, 2010 IBM Almaden Research
August 24, 2010 HP Labs Palo Alto
August 30, 2010 SAP Labs Palo Alto
September 1, 2010 Greenplum (See Florian Waas' Blog about the presentation)
September 3, 2010 Oracle Redwood Shores
September 13, 2010 Keynote at the VLDB BIRTE Workshop
September 30, 2010 IBM DB2 Community Meeting, Böblingen
October 1, 2010 SAP Walldorf
March 3, 2011 BTW 2011, Presentation
April 12, 2011 ICDE 2011, Poster
May 30, 2011 Humboldt Universität Berlin
June, 2011 "Grundlagen von Datenbanken" Workshop (Tirol, Austria)
October 26, 2011 HyPer-sonic: Combined Transaction AND Query Processing, HPTS 2011, Slides
November 18, 2011 Skalierbarkeit ODER Virtualisierung at FGDB Herbsttreffen, Potsdam
December 2, 2011 HyPer-sonic Combined Transaction AND Query Processing at HIPERFIT Workshop, Kopenhagen
June 13, 2012 Oracle Labs Research – Tea Time Talk
June 20, 2012 HyPer and its Scale-Out at Software AG
October 11, 2012 IBM DB2 Community Meeting, Böblingen
November 2, 2012 GI FG-DB Workshop Scalable Analytics
January 4, 2013 Join Processing and Indexing in Multi-Core Main-Memory Databases, Oracle Labs
March 11–15, 2013 ScyPer: A Hybrid OLTP&OLAP Distributed Main Memory Database System for Scalable Real-Time Analytics (Poster), BTW
April 5, 2013 The Adaptive Radix Tree, University of Sydney
June 10, 2013 2. Deutsches Community Treffen für GPUs in Datenbanken, TU Ilmenau
July 15, 2013 Microsoft Research Faculty Summit 2013, Slides (Thomas Neumann)
August 16, 2013 IBM Almaden Research
September 24, 2013 Hardware Transactional Memory on Haswell, HPTS 2013
Januar 31, 2014 HyPer: one DBMS for all, New England Database Summit 2014, Slides (pdf), Slides (Keynote '09), Abstract
July, 2014 Query Processing in HyPer, Cloudera
August, 2014 Query Processing in HyPer, IBM Almaden Research
March, 2015 Unnesting Arbitrary Queries, BTW Conference, Hamburg
August, 2015 How Good Are Query Optimizers, Really?, Microsoft Research, Redmond, WA
November, 2015 HyPer: The all-in-one Database System, Birthday Colloquium for Prof. Peter Lockemann, Karlsruhe Institute of Technology (KIT)
February, 2016 HyPer on Cloud 9, Workshop: Databases in the Cloud—What is different?

Summary

The HyPer prototype demonstrates that it is indeed possible to build a main-memory database system that achieves world-record transaction processing throughput and best-of-breed OLAP query response times in one system in parallel on the same database state. The two workloads of online transaction processing (OLTP) and online analytical processing (OLAP) present different challenges for database architectures. Currently, users with high rates of mission-critical transactions have split their data into two separate systems, one database for OLTP and one so-called data warehouse for OLAP. While allowing for decent transaction rates, this separation has many disadvantages including data freshness issues due to the delay caused by only periodically initiating the Extract Transform Load-data staging and excessive resource consumption due to maintaining two separate information systems. We present an efficient hybrid system, called HyPer, that can handle both OLTP and OLAP simultaneously by using an efficient multi-version concurrency control scheme. HyPer is a main-memory database system that guarantees the full ACID properties for OLTP transactions. HyPer achieves both at the same time: unprecedentedly high transaction rates as high as 100,000 per second and very fast OLAP query response times on a single system executing both workloads in parallel. The performance analysis is based on a combined TPC-C and TPC-H benchmark. HyPer's OLTP throughput is better than VoltDB's published TPC-C performance and HyPer's OLAP query response times are superior to MonetDB's query response times. It should be emphasized that HyPer can match (or beat) these two best- of-breed transaction (VoltDB) and query (MonetDB) processing engines at the same time by performing both workloads in parallel on the same database state. HyPer's performance is due to the following design:


Contact

Contact us (see Team for emails) if you are interested in a thesis, student job or even a Ph.D. position!

Technische Universität München
Institut für Informatik
Lehrstuhl III: Datenbanksysteme (I3)
Boltzmannstraße 3
85748 Garching bei München
Germany