Welcome to KMR (K Map-Reduce)#

This is KMR (K Map-Reduce), a high-performance map-reduce library. KMR-1.0 is available on the K Computer in "/opt/aics/kmr" now (2013-04-26).

KMR is a set of high-performance map-reduce operations in the MPI (Message Passing Interface) environment. It is to make programming for data-processing much easier by hiding low-level details of message passing. Its main target is large-scale computers with thousands nodes, K and Fujitsu FX10. On those platforms, KMR provides utilities for the map-reduce operations to address issues such as accessing large file-systems. But, KMR works on clusters as well.

KMR assumes large amount of memory and designed to work on-memory, whereas most map-reduce implementations are designed to work with external (disk-based) operations. So, data exchanges in KMR occur as message passing instead of remote file operations. The KMR routines work in bulk-synchronous and the most part of the code is sequential, but the code inside the mapper and reducer are multi-threaded.


Issue Tracker#



KMR comes with ABSOLUTELY NO WARRANTY. This wiki also comes with ABSOLUTELY NO WARRANTY. Contents are liable to change.


KMR is the product of AICS, Riken. Part of the results is obtained by using K Computer at AICS, RIKEN.

Are you lost?, for wiki on mt65 try: https://mt65.aics.riken.jp/wiki/