This page (revision-136) was last changed on 24-Nov-2020 16:59 by kmrdev

This page was created on 09-Mar-2018 02:16 by UnknownAuthor

Only authorized users are allowed to rename pages.

Only authorized users are allowed to delete pages.

Page revision history

Version Date Modified Size Author Changes ... Change note
136 24-Nov-2020 16:59 2 KB kmrdev to previous
135 24-Nov-2020 16:58 2 KB kmrdev to previous | to last
134 24-Nov-2020 16:55 2 KB kmrdev to previous | to last
133 19-Mar-2020 23:30 2 KB kmrdev to previous | to last
132 19-Mar-2020 23:29 2 KB kmrdev to previous | to last
131 19-Mar-2020 23:28 2 KB kmrdev to previous | to last
130 19-Mar-2020 23:27 2 KB kmrdev to previous | to last
129 19-Mar-2020 23:25 2 KB kmrdev to previous | to last
128 14-Feb-2020 17:16 3 KB kmrdev to previous | to last
127 14-Feb-2020 17:15 3 KB kmrdev to previous | to last
126 23-Aug-2018 21:05 3 KB kmrdev to previous | to last
125 23-Aug-2018 19:45 3 KB kmrdev to previous | to last
124 23-Aug-2018 19:09 3 KB kmrdev to previous | to last
123 23-Aug-2018 19:03 3 KB kmrdev to previous | to last
122 10-Jun-2018 18:03 3 KB kmrdev to previous | to last
121 10-Jun-2018 17:55 3 KB kmrdev to previous | to last

Page References

Incoming links Outgoing links

Version management

Difference between version and

At line 1 changed one line
!!! Welcome to KMR (K Map-Reduce)
!!! Welcome to KMR
At line 3 changed 2 lines
This is KMR (K Map-Reduce), a high-performance map-reduce library.
KMR-1.0 is available on K Computer in "/opt/aics/kmr" now (2013-04-26).
This is KMR, a high-performance map-reduce library.
KMR-1.0 is available since 2013-04-26.
KMR works on ordinary clusters as well as large-scale supercomputers.
KMR source code is available under the BSD license.
At line 6 changed one line
KMR is a set of high-performance map-reduce operations in the MPI (Message Passing Interface) environment. It is targeted to large-scale computers with thousands nodes, especially to ones such as Fujitsu K and FX10. KMR works on clusters as well.
__Latest release is KMR-1.10 (2018-11-16)__.
At line 8 changed one line
KMR assumes large amount of memory and designed to work on-memory, whereas most map-reduce implementations are designed to work with external (disk-based) operations. So, data exchanges in KMR occur as message passing instead of remote file operations. The KMR routines work in bulk-synchronous and the most part of the code is single-threaded, but the code inside the mapper and reducer includes compiler directives for OpenMP threading.
KMR is a set of high-performance map-reduce operations in the MPI (Message Passing Interface) environment.
It makes programming for data-processing much easier by hiding low-level details of message passing.
Its main targets are large-scale supercomputers with thousands of compute nodes.
%%strike KMR provides utilities other than map-reduce operations to address issues such as accessing very large file-systems, on platforms K and Fujitsu FX10/%.
At line 10 changed 6 lines
* Overview and API Document:
** [http://mt65.aics.riken.jp/kmrdoc/kmr-1.0/html/index.html]]
** It is a Doxgen Generated document.
* KMR Issue Tracker:
** [https://mt65.aics.riken.jp/jtrac/]
** Please make a new user by the login-page to report a new issue.
KMR is designed to work in-memory and to exploit large amount of memory available on supercomputers, whereas most map-reduce implementations are designed to work with external (disk-based) operations.
So, data exchanges in KMR occur as message passing instead of remote file operations.
The KMR routines work in bulk-synchronous and the most part of the code is sequential, but the code inside the mapper and reducer are multi-threaded.
At line 17 changed 2 lines
* Download:
** [https://mt65.aics.riken.jp/kmr]
!!Documents
At line 20 changed 4 lines
* Project Overview and other Activities of the Team:
** [http://mt.aics.riken.jp]
* Overview and API Document (newer, corrected, for next release):
** [http://mt65.aics.riken.jp/kmrdoc/newest/html/index.html]]
* [Overview and API Document|https://riken-rccs.github.io/kmr/]
** It is a Doxgen generated document, included in the installation.
!!Downloading
* [Download]
!!Tutorials
* [Tutorial] (in Japanese)
!!Project Site
* KMR in GitHub [https://github.com/riken-rccs]
* Issue reporting [https://github.com/riken-rccs/kmr/issues]
* Other software from RIKEN R-CCS [https://riken-rccs.github.io]
!!Publications
* __K MapReduce: A Scalable Tool for Data-Processing and Search/Ensemble Applications on Large-Scale Supercomputers__. Motohiko Matsuda, Naoya Maruyama, and Shinichiro Takizawa. IEEE Cluster Computing (CLUSTER) 2013. (C) Copyright IEEE. [ieeexplore.ieee.org|https://ieeexplore.ieee.org/document/6702663]\\
It describes an overview and optimizations used in KMR.
* __Supporting Workflow Management of Scientific Applications by MapReduce Programming Model__. Shinichiro Takizawa, Motohiko Matsuda, and Naoya Maruyama. IPSJ HPCS 2014. (in Japanese). [http://id.nii.ac.jp/1001/00096874]\\
It describes some scientific applications workflow implemented in MapReduce using KMR.
* __Evaluation of Asynchronous MPI Communication in Map-Reduce System on the K Computer__. Motohiko Matsuda, Naoya Maruyama, and Shinichiro Takizawa. EuroMPI Workshop 2014. (C) Copyright ACM. [dl.acm.org|https://dl.acm.org/doi/10.1145/2642769.2642800]\\
It compares all-to-all collective communication versus asynchronous communication in shuffling communication, to qualify believed effectiveness of overlapping of communication and computation.
At line 51 added 7 lines
!!Acknowledgment
KMR is a product of RIKEN R-CCS.
Part of the results is obtained by using K computer at RIKEN R-CCS.
----
At line 31 removed 5 lines
----
\\
Are you lost?, for wiki on mt65 try: [https://mt65.aics.riken.jp/wiki/]