

<meta name="robots" content="noindex,nofollow" />
<link rel="shortcut icon" type="image/x-icon" href="/kmr/images/favicon.ico" />

<link rel="icon" type="image/x-icon" href="/kmr/images/favicon.ico" />


<pre>
!!! Welcome to KMR

This is KMR, a high-performance map-reduce library.
KMR-1.0 is available on the K computer since 2013-04-26.
KMR works on ordinary clusters as well.

__Latest release is KMR-1.2 (2014-06-27)__.

KMR is a set of high-performance map-reduce operations in the MPI (Message Passing Interface) environment.
It makes programming for data-processing much easier by hiding low-level details of message passing.
Its main targets are large-scale supercomputers with thousands of compute nodes, such as K and Fujitsu FX10.
On these platforms, KMR provides utilities other than the map-reduce operations which address issues such as accessing very large file-systems.

KMR is designed to work on-memory and exploits large amount of memory available on supercomputers, whereas most map-reduce implementations are designed to work with external (disk-based) operations.
So, data exchanges in KMR occur as message passing instead of remote file operations.
The KMR routines work in bulk-synchronous and the most part of the code is sequential, but the code inside the mapper and reducer are multi-threaded.

!!Project Site

* Project Overview and other Activities of the Team at RIKEN:
** [http://mt.aics.riken.jp]

!!Documents

* Overview and API Document:
** [http://mt.aics.riken.jp/kmr/docs/kmr-1.2/html/index.html]
** It is a Doxgen generated document, included in the installation.
//* Overview and API Document (Newer, corrected, for the next release):
//** [https://mt.aics.riken.jp/kmr/docs/next/html/index.html]
//** Documentation in the distribution is late, and it is placed here for late breaking.
* Tutorial:
** KMR Tutorial in AICS Software Seminar on May 20, 2014 (Japanese)
*** [Handout|Main/tutorial140520.pdf], [Q&amp;A List|Main/tutorialQaA140520.pdf]
*** [Handout(supplementary)|Main/tutorial140520-2.pdf]

!!Downloading

* Source Code Download:
** [Download]
** KMR source is available with LGPL-2.1.

!!Issue Reporting

* KMR Issue Tracker:
** [http://mt.aics.riken.jp/jtrac/]
** Please make a new user by the login-page to report a new issue.

!!Publications

* [cluster2013.pdf]: __K MapReduce: A Scalable Tool for Data-Processing and Search/Ensemble Applications on Large-Scale Supercomputers__.  Motohiko Matsuda, Naoya Maruyama, and Shinichiro Takizawa. IEEE Cluster Computing (CLUSTER) 2013. (C) Copyright IEEE.  [IEEE Explore|http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6702663].\\
 It describes an overview and optimizations used in KMR.

* [hpcs2014.pdf]: __Supporting Workflow Management of Scientific Applications by MapReduce Programming Model__. Shinichiro Takizawa, Motohiko Matsuda, and Naoya Maruyama. IPSJ HPCS 2014. (Japanese)\\
 It describes some scientific applications workflow implemented in MapReduce using KMR.

* [bigdata2014.pdf]: __Evaluation of Asynchronous MPI Communication in Map-Reduce System on the K Computer__.  Motohiko Matsuda, Naoya Maruyama, and Shinichiro Takizawa. EuroMPI Workshop 2014.  (C) Copyright ACM.  [ACM DL|http://dl.acm.org/citation.cfm?id=2642800]\\
 It compares two implementations of shuffling communication, all-to-all collective v.s. point-to-point asynchronous communication.

----

!!DISCLAIMER

KMR comes with ABSOLUTELY NO WARRANTY.
This wiki also comes with ABSOLUTELY NO WARRANTY.
Contents are liable to change.

----

!!Acknowledgment

KMR is a product of RIKEN AICS.
Part of the results is obtained by using the K computer at RIKEN AICS.

----

</pre>