Welcome to KMR#
This is KMR, a high-performance map-reduce library. KMR-1.0 is available on the K computer since 2013-04-26. KMR works on ordinary clusters as well.
Latest release is KMR-1.1 (2013-09-20)..
KMR is a set of high-performance map-reduce operations in the MPI (Message Passing Interface) environment. It makes programming for data-processing much easier by hiding low-level details of message passing. Its main targets are large-scale supercomputers with thousands of compute nodes, such as K and Fujitsu FX10. On these platforms, KMR provides utilities other than the map-reduce operations which address issues such as accessing very large file-systems.
KMR is designed to work on-memory and exploits large amount of memory available on supercomputers, whereas most map-reduce implementations are designed to work with external (disk-based) operations. So, data exchanges in KMR occur as message passing instead of remote file operations. The KMR routines work in bulk-synchronous and the most part of the code is sequential, but the code inside the mapper and reducer are multi-threaded.
Project Site#
- Project Overview and other Activities of the Team at RIKEN:
Documents#
- Overview and API Document:
- http://mt.aics.riken.jp/kmr/docs/kmr-1.1/html/index.html
- It is a Doxgen generated document, included in the installation.
- http://mt.aics.riken.jp/kmr/docs/kmr-1.1/html/index.html

Downloading#
- Source Code Download:
- Download
- KMR source is available with LGPL-2.1.
Issue Reporting#
- KMR Issue Tracker:
- http://mt.aics.riken.jp/jtrac/
- Please make a new user by the login-page to report a new issue.
- http://mt.aics.riken.jp/jtrac/
Publications#
- cluster2013.pdf: K MapReduce: A Scalable Tool for Data-Processing and Search/Ensemble Applications on Large-Scale Supercomputers. Motohiko Matsuda, Naoya Maruyama, and Shinichiro Takizawa. IEEE Cluster Computing (CLUSTER) 2013. (C) Copyright IEEE. IEEE Explore
.
It describes an overview and optimisations used in KMR.
DISCLAIMER#
KMR comes with ABSOLUTELY NO WARRANTY. This wiki also comes with ABSOLUTELY NO WARRANTY. Contents are liable to change.
Acknowledgment#
KMR is a product of RIKEN AICS. Part of the results is obtained by using the K computer at RIKEN AICS.