News: See you at VLDB 2014 in Hangzhou! (two presentations, see publications below)

Highlights

In-memory Data Management

HyPer relies on in-memory data management without the ballast of traditional database systems caused by DBMS-controlled page structures and buffer management. SQL table definitions are transformed into simple vector-based virtual memory representations – which constitutes a column oriented physical storage scheme.

Efficient Snapshotting

OLAP query processing is separated from mission-critical OLTP transaction processing by forking virtual memory snapshots. Thus, no concurrency control mechanisms are needed – other than the hardware-assisted transparent VM management – to separate the two workload classes.

Data-centric Code Generation

Transactions and queries are specified in SQL or a PL/SQL-like scripting language and are efficiently compiled into efficient LLVM assembly code.

No compromises

HyPer's transaction processing is fully ACID-compliant. Queries are specified in SQL-92 plus some extensions from subsequent standards.


Team

Head: Prof. Alfons Kemper, Prof. Thomas Neumann

Ph.D. Students: Robert Brunel, Jan Finis, Florian Funke, Moritz Kaufmann, Viktor Leis, Tobias Mühlbauer, Wolf Rödiger, Manuel Then


Publications

Foundational HyPer publications

Venue Publication Link
ICDE 2011 HyPer: A Hybrid OLTP&OLAP Main Memory Database System Based on Virtual Memory Snapshots
Alfons Kemper and Thomas Neumann, 2011.
IEEE Xplore
Report (pdf)
VLDB 2011 Efficiently Compiling Efficient Query Plans for Modern Hardware
Thomas Neumann, 2011.
pdf

List of all publications

Venue Publication Link
Datenbank Spektrum HyPer Beyond Software: Exploiting Modern Hardware for Main-Memory Database Systems
Florian Funke, Alfons Kemper, Tobias Mühlbauer, Thomas Neumann, Viktor Leis, 2014.
Springer DL
VLDB 2014 Engineering High-Performance Database Engines
Thomas Neumann, 2014.
pdf
DaMoN 2014 Heterogeneity-Conscious Parallel Query Execution: Getting a better mileage while driving faster!
Tobias Mühlbauer, Wolf Rödiger, Robert Seilbeck, Alfons Kemper, Thomas Neumann, 2014.
pdf
SIGMOD 2014 Morsel-Driven Parallelism: A NUMA-Aware Query Evaluation Framework for the Many-Core Age
Viktor Leis, Peter Boncz, Alfons Kemper, Thomas Neumann, 2014.
pdf
SIGMOD 2014 One DBMS for all: the Brawny Few and the Wimpy Crowd (Demonstration)
Tobias Mühlbauer, Wolf Rödiger, Robert Seilbeck, Angelika Reiser, Alfons Kemper, Thomas Neumann, 2014.
pdf
DEBULL Compiling Database Queries into Machine Code
Thomas Neumann, Viktor Leis, Data Engineering Bulletin, March 2014.
pdf
ICDE 2014 Locality-Sensitive Operators for Parallel Main-Memory Database Clusters
Wolf Rödiger, Tobias Mühlbauer, Philipp Unterbrunner, Angelika Reiser, Alfons Kemper, Thomas Neumann, 2014.
pdf
ICDE 2014 Exploiting Hardware Transactional Memory in Main-Memory Databases
Viktor Leis, Alfons Kemper, Thomas Neumann, 2014. Best Paper Award
pdf
PVLDB 2013, VLDB 2014 Instant Loading for Main Memory Databases
Tobias Mühlbauer, Wolf Rödiger, Robert Seilbeck, Angelika Reiser, Alfons Kemper, Thomas Neumann, 2013.
pdf
CIKM 2013 RWS-Diff: Flexible and Efficient Change Detection in Hierarchical Data
Jan Finis, Martin Raiber, Nikolaus Augsten, Robert Brunel, Alfons Kemper, Franz Färber, 2013.
IMDM 2013 An Evaluation of Strict Timestamp Ordering Concurrency Control for Main-Memory Database Systems
Stephan Wolf, Henrik Mühe, Alfons Kemper, Thomas Neumann, 2013.
IMDM 2013 Massively Parallel NUMA-aware Hash Joins
Harald Lang, Viktor Leis, Martina-Cezara Albutiu, Thomas Neumann, Alfons Kemper, 2013.
DEBULL Transaction Processing in the Hybrid OLTP&OLAP Main-Memory Database System HyPer
Alfons Kemper, Thomas Neumann, Jan Finis, Florian Funke, Viktor Leis, Henrik Mühe, Tobias Mühlbauer, Wolf Rödiger, IEEE Computer Society Data Engineering Bulletin, Special Issue on "Main Memory Databases", 2013.
Issue
DanaC 2013 ScyPer: Elastic OLAP Throughput on Transactional Data
Tobias Mühlbauer, Wolf Rödiger, Angelika Reiser, Alfons Kemper, Thomas Neumann, 2013.
ACM DL
SIGMOD 2013 DeltaNI: An Efficient Labeling Scheme for Versioned Hierarchical Data
Jan Finis, Robert Brunel, Alfons Kemper, Thomas Neumann, Franz Faerber, Norman May, 2013.
BTW 2013 Extending the MPSM Join
Martina-Cezara Albutiu, Alfons Kemper, Thomas Neumann, 2013.
pdf
BTW 2013 ScyPer: A Hybrid OLTP&OLAP Distributed Main Memory Database System for Scalable Real-Time Analytics (Demonstration)
Tobias Mühlbauer, Wolf Rödiger, Angelika Reiser, Alfons Kemper, Thomas Neumann, 2013.
pdf
CIDR 2013 Executing Long-Running Transactions in Synchronization-Free Main Memory Database Systems
Henrik Mühe and Alfons Kemper and Thomas Neumann, 2013.
pdf
ICDE 2013 CPU and Cache Efficient Management of Memory-Resident Databases
Holger Pirk, Florian Funke, Martin Grund, Thomas Neumann, Ulf Leser, Stefan Manegold, Alfons Kemper, Martin Kersten, 2013.
ICDE 2013 The Adaptive Radix Tree: ARTful Indexing for Main-Memory Databases
Viktor Leis, Alfons Kemper and Thomas Neumann, 2013.
pdf
VLDB 2012 Massively Parallel Sort-Merge Joins in Main Memory Multi-Core Database Systems
Martina-Cezara Albutiu, Alfons Kemper and Thomas Neumann, 2012.
pdf
VLDB 2012 Compacting Transactional Data in Hybrid OLTP&OLAP Databases
Florian Funke, Alfons Kemper, Thomas Neumann, 2012.
pdf
DEBULL HyPer: Adapting Columnar Main -Memory Data Management for Transactional AND Query Processing
Alfons Kemper, Thomas Neumann, Florian Funke, Viktor Leis, Henrik Mühe, Bulletin of the Technical Committee on Data Engineering, March 2012, Vol. 35, No. 1, pp. 46–51.
Issue
Technical Report Massively Parallel Sort-Merge Joins in Main Memory Multi-Core Database Systems
Martina-Cezara Albutiu, Alfons Kemper and Thomas Neumann, Technical Report, TUM-I121, March, 16, 2012.
pdf, pptx
EDBT 2012 The Mainframe Strikes Back: Elastic Multi-Tenancy Using Main Memory Database Systems On A Many-Core Server
Henrik Mühe, Alfons Kemper and Thomas Neumann, 2012.
VLDB 2011 HyPer-sonic Combined Transaction AND Query Processing
Florian Funke and Alfons Kemper and Thomas Neumann, 2011.
VLDB 2011 Efficiently Compiling Efficient Query Plans for Modern Hardware
Thomas Neumann, 2011.
pdf
DBTest 2011 The mixed workload CH-benCHmark
Dagstuhl "Robust Query Processing" Breakout Group "Workload Management", 2011.
DaMoN 2011 How to Efficiently Snapshot Transactional Data: Hardware or Software Controlled?
Henrik Mühe and Alfons Kemper and Thomas Neumann, 2011.
Datenbank Spektrum HyPer: Die effiziente Reinkarnation des Schattenspeichers in einem Hauptspeicher-DBMS
Florian Funke and Alfons Kemper and Henrik Mühe and Thomas Neumann, 2011.
Datenbank Spektrum
BTW 2011 Benchmarking Hybrid OLTP&OLAP Database Systems
Florian Funke and Alfons Kemper and Thomas Neumann, 2011.
ICDE 2011 HyPer: A Hybrid OLTP&OLAP Main Memory Database System Based on Virtual Memory Snapshots
Alfons Kemper and Thomas Neumann, 2011.
IEEE Xplore
Technical Report HyPer - Hybrid OLTP&OLAP High Performance Database System
Alfons Kemper and Thomas Neumann, Technical Report, TUM-I1010, May, 19, 2010.
pdf

Presentations/Mentions

Date Venue
May 21, 2010 Colloquium of the Chair of Database Systems
May 26, 2010 "Grundlagen von Datenbanken " Workshop (GvDB, Bad Helmstedt)
June 26, 2010 IBM Böblingen
July 22, 2010 Inaugural Lecture ("Antrittsvorlesung" Thomas Neumann)
August 13, 2010 IBM Almaden Research
August 24, 2010 HP Labs Palo Alto
August 30, 2010 SAP Labs Palo Alto
September 1, 2010 Greenplum (See Florian Waas' Blog about the presentation)
September 3, 2010 Oracle Redwood Shores
September 13, 2010 Keynote at the VLDB BIRTE Workshop
September 30, 2010 IBM DB2 Community Meeting, Böblingen
October 1, 2010 SAP Walldorf
March 3, 2011 BTW 2011, Presentation
April 12, 2011 ICDE 2011, Poster
May 30, 2011 Humboldt Universität Berlin
June, 2011 "Grundlagen von Datenbanken" Workshop (Tirol, Austria)
October 26, 2011 HyPer-sonic: Combined Transaction AND Query Processing, HPTS 2011, Slides
November 18, 2011 Skalierbarkeit ODER Virtualisierung at FGDB Herbsttreffen, Potsdam
December 2, 2011 HyPer-sonic Combined Transaction AND Query Processing at HIPERFIT Workshop, Kopenhagen
June 13, 2012 Oracle Labs Research – Tea Time Talk
June 20, 2012 HyPer and its Scale-Out at Software AG
October 11, 2012 IBM DB2 Community Meeting, Böblingen
November 2, 2012 GI FG-DB Workshop Scalable Analytics
January 4, 2013 Join Processing and Indexing in Multi-Core Main-Memory Databases, Oracle Labs
March 11–15, 2013 ScyPer: A Hybrid OLTP&OLAP Distributed Main Memory Database System for Scalable Real-Time Analytics (Poster), BTW
April 5, 2013 The Adaptive Radix Tree, University of Sydney
June 10, 2013 2. Deutsches Community Treffen für GPUs in Datenbanken, TU Ilmenau
July 15, 2013 Microsoft Research Faculty Summit 2013, Slides (Thomas Neumann)
August 16, 2013 IBM Almaden Research
September 24, 2013 Hardware Transactional Memory on Haswell, HPTS 2013
Januar 31, 2014 HyPer: one DBMS for all, New England Database Summit 2014, Slides (pdf), Slides (Keynote '09), Abstract

Summary

The HyPer prototype demonstrates that it is indeed possible to build a main-memory database system that achieves world-record transaction processing throughput and best-of-breed OLAP query response times in one system in parallel on the same database state. The two workloads of online transaction processing (OLTP) and online analytical processing (OLAP) present different challenges for database architectures. Currently, users with high rates of mission-critical transactions have split their data into two separate systems, one database for OLTP and one so-called data warehouse for OLAP. While allowing for decent transaction rates, this separation has many disadvantages including data freshness issues due to the delay caused by only periodically initiating the Extract Transform Load-data staging and excessive resource consumption due to maintaining two separate information systems. We present an efficient hybrid system, called HyPer, that can handle both OLTP and OLAP simultaneously by using hardware-assisted replication mechanisms to maintain consistent snapshots of the transactional data. HyPer is a main-memory database system that guarantees the full ACID properties for OLTP transactions and executes OLAP query sessions (multiple queries) on arbitrarily current and consistent snapshots. The utilization of the processor-inherent support for virtual memory management (address translation, caching, copy-on-write) yields both at the same time: unprecedentedly high transaction rates as high as 100000 per second and very fast OLAP query response times on a single system executing both workloads in parallel. The performance analysis is based on a combined TPC-C and TPC-H benchmark. We have developed the novel hybrid OLTP&OLAP database system HyPer that is based on snapshotting transactional data via the virtual memory management of the operating system. In this architecture the OLTP process owns the database and periodically (e.g., in the order of seconds or minutes) forks an OLAP process. This OLAP process constitutes a fresh transaction consistent snapshot of the database. Thereby, we exploit operating systems functionality to create virtual memory snapshots for new, cloned processes. In Unix, for example, this is done by creating a child process of the OLTP process via the fork system call. The forked child process obtains an exact copy of the parent processes address space. This virtual memory snapshot that is created by the fork-operation will be used for executing a session of OLAP queries. These queries can be executed in parallel threads or serially, depending on the system resources or client requirements. In essence, the virtual memory snapshot mechanism constitutes a OS/hardware supported shadow paging mechanism as proposed decades ago for disk-based database systems. However, the original proposal incurred severe costs as it had to be software-controlled and it destroyed the clustering on disk. Neither of these drawbacks occurs in the virtual memory snapshotting as clustering across RAM pages is not an issue. Furthermore, the sharing of pages and the necessary copy-on-update/write is managed by the operating system with effective hardware support of the MMU (memory management unit) via the page table that translates VM addresses to physical pages and traps necessary replication (copy-on-write) actions. Therefore, the page replication is extremely efficiently done in 2μs as we measured in a micro-benchmark. HyPer's OLTP throughput is better than VoltDB's published TPC-C performance and HyPer's OLAP query response times are superior to MonetDB's query response times. It should be emphasized that HyPer can match (or beat) these two best- of-breed transaction (VoltDB) and query (MonetDB) processing engines at the same time by performing both workloads in parallel on the same database state. HyPer's performance is due to the following design:


Contact

Contact us (see Team for emails) if you are interested in a thesis, student job or even a Ph.D. position!

Technische Universität München
Institut für Informatik
Lehrstuhl III: Datenbanksysteme (I3)
Boltzmannstraße 3
85748 Garching bei München
Germany