MapReduce: Simplified Data Processing on Large Clusters

Project Overview

Documentation

• Libary

  • labgob
  • labrpc
  • logger
  • deepcopy

• Architecture

  • mapreduce
  • raft

• Application

  • kvraft

• Others

  • configs

Algorithms

Project‘s Catalogue:

.
|-- mr // mapreduce libary
|-- mrapp // application for mapreduce
|-- main // start up a mapreduce
|-- raft // raft libary
|-- kvraft // kv service
|-- labrpc // rpc libary
|-- labgob // gob libary
|-- config // hyper-parameter
|-- utils // logger, deepcopy,..
`-- script // dstest, dslogs

MapReduce

A. Progress

• worker count test. PASS
• job count test. PASS
• early exit test. PASS
• indexer test. PASS
• map parallelism test. PASS
• reduce parallelism test. PASS
• crash test. NO

B. Debug

log.Pirntln("...")
log.Printf("...")

Reference

• MIT 6.5840 Distributed System
• GO