Skip to content

MapReduce architecture design #100

@hongchaodeng

Description

@hongchaodeng

I had a brief talk with MR folks about the design. Here's the basic thoughts:

  • Master make all decisions.
  • Mapper writes output to local disk.
  • Reducer and shuffler are on the same node.
  • Shuffler reads data from all mappers (this could be optimized by mapper local combier) and shuffles. Reducer reads shuffled result key by key.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions