![Homework1
People You Might Know
First Mapper
Read each line to generate friend pairs
Parameters: key- friend_pair
If already friend, value = 0
If 1 common friend, value = 1
E.g. {1: (2, 4)} [((1, 2), 0), ((1,4), 0), ((2, 4), 1)]
First Reducer
Count total common friends for each key
Parameters: key - friend_pair
value – sum(value)
E.g. {(2, 4): [1, 1, 1, 0,1,0]} {(2, 4): 4}](https://image.slidesharecdn.com/mapreduce-150822074726-lva1-app6891/85/How-To-Run-Mapreduce-Jobs-In-Python-6-320.jpg)
![Homework1
People You Might Know
Second Mapper
Use user as key to map the value pairs
Parameters: key – user
value – friends, sum(value)
E.g. {(2, 4): 4} {2: (4, 4)}
Second Reducer
For each user, find the most 10th common friend
Parameters: key – user
value – the most 10th common friends
E.g. {2: [<4, 4>, <5, 11>, <6, 10>….]} {2: [5, 6, …]}](https://image.slidesharecdn.com/mapreduce-150822074726-lva1-app6891/85/How-To-Run-Mapreduce-Jobs-In-Python-7-320.jpg)
![Homework1
People You Might Know
Multi-step jobs
Use MRJob.steps(self) to connect mrjob
pipeline
def steps(self):
return [
MRStep(mapper = self.mapper1,
reducer = self.reducer1),
MRStep(mapper = self.mapper2,
reducer=self.reducer2)
]](https://image.slidesharecdn.com/mapreduce-150822074726-lva1-app6891/85/How-To-Run-Mapreduce-Jobs-In-Python-8-320.jpg)
How To Run Mapreduce Jobs In Python
- 2. mrjob: Python Mapreduce Library Some Important Features: mrjob helps you to write MapReduce jobs in Python and run them on Hadoop mrjob also allows you to run test code locally without installing Hadoop mrjob can write multi-step MapReduce jobs (eg.CS246 Homework1-Questions1) More Information: https://github.com/Yelp/mrjob
- 3. Run Hadoop W/O VM OSX platform brew install hadoop pip install mrjob Change configs core-site.xml hdfs-site.xml mapred-site.xml yarn-site.xml More detailed information: http://stackoverflow.com/questions/25358793/error- launching-job-using-mrjob-on-hadoop
- 4. Example: Word Count Run locally python WordCount.py read.txt Run on Hadoop python WordCount.py read.txt –r hadoop Python vs. Java Line: 13 vs. 61
- 5. Homework1 People You Might Know Write a MapReduce program in Hadoop that implements a simple “People You Might Know” social network friendship recommendation algorithm. The key idea is that if two people have a lot of mutual friends, then the system should recommend that they connect with each other. Input: <User><TAB><Friends> Output: <User><TAB><Recommendations>
- 6. Homework1 People You Might Know First Mapper Read each line to generate friend pairs Parameters: key- friend_pair If already friend, value = 0 If 1 common friend, value = 1 E.g. {1: (2, 4)} [((1, 2), 0), ((1,4), 0), ((2, 4), 1)] First Reducer Count total common friends for each key Parameters: key - friend_pair value – sum(value) E.g. {(2, 4): [1, 1, 1, 0,1,0]} {(2, 4): 4}
- 7. Homework1 People You Might Know Second Mapper Use user as key to map the value pairs Parameters: key – user value – friends, sum(value) E.g. {(2, 4): 4} {2: (4, 4)} Second Reducer For each user, find the most 10th common friend Parameters: key – user value – the most 10th common friends E.g. {2: [<4, 4>, <5, 11>, <6, 10>….]} {2: [5, 6, …]}
- 8. Homework1 People You Might Know Multi-step jobs Use MRJob.steps(self) to connect mrjob pipeline def steps(self): return [ MRStep(mapper = self.mapper1, reducer = self.reducer1), MRStep(mapper = self.mapper2, reducer=self.reducer2) ]
- 9. Homework1 People You Might Know Run locally python friends_commend.py soc- LiveJournal1Adj-2.txt > friends.txt Run on Hadoop python friends_commend.py soc- LiveJournal1Adj-2.txt –r hadoop > friends.txt
- 10. References Stanford CS246: Mining Massive Data Sets (Winter 2015). (n.d.). Retrieved August 22, 2015. Yelp/mrjob. (n.d.). Retrieved August 22, 2015. Error launching job using mrjob on Hadoop. (n.d.). Retrieved August 22, 2015. MapReduce Tutorial. (n.d.). Retrieved August 22, 2015.