This is version 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 . It is not the current version, and thus it cannot be edited.

Back to current version Restore this version

Welcome to KMR#

This is KMR, a high-performance map-reduce library. KMR-1.0 is available on the K computer since 2013-04-26. KMR works on ordinary clusters as well.

Latest release is KMR-1.1 (2013-09-20)..

KMR is a set of high-performance map-reduce operations in the MPI (Message Passing Interface) environment. It makes programming for data-processing much easier by hiding low-level details of message passing. Its main targets are large-scale supercomputers with thousands of compute nodes, such as K and Fujitsu FX10. On these platforms, KMR provides utilities other than the map-reduce operations which address issues such as accessing very large file-systems.

KMR is designed to work on-memory and exploits large amount of memory available on supercomputers, whereas most map-reduce implementations are designed to work with external (disk-based) operations. So, data exchanges in KMR occur as message passing instead of remote file operations. The KMR routines work in bulk-synchronous and the most part of the code is sequential, but the code inside the mapper and reducer are multi-threaded.

Project Site#

• Project Overview and other Activities of the Team at RIKEN:
  • http://mt.aics.riken.jp

Documents#

• Overview and API Document:
  • http://mt.aics.riken.jp/kmr/docs/kmr-1.1/html/index.html
  • It is a Doxgen generated document, included in the installation.

//* Overview and API Document (Newer, corrected, for the next release): //** https://mt.aics.riken.jp/kmr/docs/next/html/index.html //** Documentation in the distribution is late, and it is placed here for late breaking.

Downloading#

• Source Code Download:
  • Download
  • KMR source is available with LGPL-2.1.

Issue Reporting#

• KMR Issue Tracker:
  • http://mt.aics.riken.jp/jtrac/
  • Please make a new user by the login-page to report a new issue.

Publications#

• cluster2013.pdf: K MapReduce: A Scalable Tool for Data-Processing and Search/Ensemble Applications on Large-Scale Supercomputers. Motohiko Matsuda, Naoya Maruyama, and Shinichiro Takizawa. IEEE Cluster Computing (CLUSTER) 2013. (C) Copyright IEEE. IEEE Explore.
  It describes an overview and optimisations used in KMR.

---

DISCLAIMER#

KMR comes with ABSOLUTELY NO WARRANTY. This wiki also comes with ABSOLUTELY NO WARRANTY. Contents are liable to change.

---

Acknowledgment#

KMR is a product of RIKEN AICS. Part of the results is obtained by using the K computer at RIKEN AICS.

---