Page rank talk at NTU-EE

Ping Yeh
Google, Inc.
2010/12/20
Department of Electric Engineering, NTU
The PageRank algorithm and
some selected applications

Disclaimer (legal)
The content of this talk is the speaker's personal opinion and
is not the opinion or policy of his employer.

Outline
● Introduction
● Google and Google search
● PageRank algorithm for ranking web pages
● Using MapReduce to calculate PageRank for billions of
pages
● Impact factor of journals and PageRank
● Conclusion

Google
● The name: homophone to the word “Googol” which means
10100
.
● The company:
○ founded by Larry Page and Sergey Brin in 1998,
○ ~20,000 employees as of 2009,
○ spread in 68 offices around the world (23 in N. America, 3 in
Latin America, 14 in Asia Pacific, 23 in Europe, 5 in Middle
East and Africa).
● The mission: “to organize the world's information and make
it universally accessible and useful.”

Google Services
Photo by mr.hero on panoramio (http://www.panoramio.com/photo/1127015)
news
book
search
picasaweb
product
search
video
maps
Gmail
Android
blogger.com
groups
calendar
scholar
Earth
Sky
web search
desktop
translate
talk
YouTube
Chrome
reader
iGoogle

Google Search
● http://www.google.com/ or http://www.google.com.tw/

The abundance problem
Quote Langville and Meyer's nice book “Google's PageRank
and beyond: the science of search engine rankings”:
The men in Jorge Luis Borges’ 1941 short story, “The Library
of Babel”, which describes an imaginary, infinite library.
● When it was proclaimed that the Library contained all books, the
ﬁrst impression was one of extravagant happiness. All men felt
themselves to be the masters of an intact and secret treasure.
There was no personal or world problem whose eloquent
solution did not exist in some hexagon.
. . . As was natural, this inordinate hope was followed by an
excessive depression. The certitude that some shelf in some
hexagon held precious books and that these precious books

The power of web search engines nowadays
Search vs. Typical desktop
document application
Data size O(1014
B) O(106
B)
Speed ~ 250 ms ~ 1s
Relevance high low
Cloud
Computing
Ranking
Web
Proliferation

Serving a query
Google Web Server
Spell checker
Ad Server
I0
I1
I2
IN
I0
I1
I2
IN
I0
I1
I2
IN
…
…
D0
D1
DM
D0
D1
DM
D0
D1
DM
…
…
IEEE Micro, 2003
query
Misc. servers
Doc serversIndex servers
Index shards Doc shards
0.25 seconds, 1000+ machines

Major contributors
● Ranking the results: Larry Page,
“For the creation of the Google search engine.”
(2004)
● Indexing the web: Sergey Brin,
“For leadership in development of rapid indexing
and retrieval of relevant information from the
World Wide Web.” (2009)
● Dealing with scales: Jeff Dean and
Sanjay Ghemawat,
“for contributions to the science and
engineering of large-scale distribute
computer systems.” (2009)

Google Scholar
● Search for scholarly literature from one place
○ Motto: “Stand on the shoulders of giants”
● http://scholar.google.com/

What can be found with Google Scholar?
● Types: articles, theses, books and abstracts.
○ Added court opinions from U.S. federal and state district,
appellate and supreme courts on Nov 17.
● Sources: academic publishers, professional societies,
online repositories, universities and other web sites.

Ranking the web
● Given a query, which are the most relevant / authoritative
pages?
● Which pages should be in the search result?
○ Find pages containing the query?
■ Not enough. By searching “NTU physics” you expect to see
http://www.phys.ntu.edu.tw/ in results, but that page contains
only one “NTU” and zero “physics” - because it's in Chinese!
○ Use links, anchor texts and other means to bring it in
● How to rank these pages?
○ Content analysis
○ Link analysis

The web as a graph
www.cnn.com
www.nytimes.com
en.wikipedia.org

Link analysis of the web graph
● Crawl web pages and form a graph G = (V, E)
○ V = vertices = nodes in the graph. One node is one web
page with a unique URL. Let N = | V |.
○ E = edges = links in the graph. One link is one hyperlink.
● Calculate and attach score/indicators/ranks to nodes based
on the links
○ in-links/out-links: links that enter/leave a node
○ in-degree/out-degree = number of in-links/out-links
● NxN Adjacency matrix A: Aij
= 1 if (i,j) ∈ E, 0 otherwise.
○ A is sparse: average out-degree ~ 10, N ~ O(1010
)

Diameter of the web by Albert et. al. (1999)
● Power-law distribution of
out-links (a) and in-links (b)
with out
= 2.45, in
= 2.1.
“Scale-free network”
● Mean distance vs. graph
size.
“Small world”

The challenges of web search
● Web is huge.
○ O(1010
) pages, O(1011
) links
○ the web graph most likely can't be stored in one computer's
memory.
● Web is dynamic. Web pages appear and disappear
everyday.
● Web is self-organized. Every author is free to create/delete
pages and make links to which ever page s/he likes. There
is no “formula” or central control

3
4
5
6
1
2
7
8
A test graph
source sink
auth
hub
source: is like a
master thesis that
nobody knows
about.
sink: is a paper
that cites nobody
hub: much more
out-links than in-
links, like a review
article.
auth: much more in-
links than out-links, like
an “authoritative” paper.

Bibliometrics
● The study of written documents and their citation structure.
● A long and profound interest in use of citations to produce
quantitative estimates of the importance and “impact” of
individual scientiﬁc papers and journals.
○ Scientific value of a paper is only as much as recognized by
the social network in the scientific community?
● The most well-known metric: Garﬁeld’s impact factor.

Garfield's Impact Factor (1972)
● Used to provide a numerical assessment of journals in
Journal Citation Reports of the Institute for Scientiﬁc
Information.
○ used in criteria for Ph.D. completion or as a reference point in
proposal review in some places.
● IF of a journal X = average number of citations per paper
received this year for articles published in X in last 2 years.

3
4
5
6
1
2
7
8
Impact Factor
0
1
4
1
2
1
5
2

Katz's Path-counting algorithm (1953)
● For evaluating the “status” of a person or a group by who
“chooses” them.
● Number of paths from person i to person j of length k =
Note that
● Contribution from person i to person j's status = a weighted
path sum. The constant b is context and group dependent.
● The “status” of person j is then the sum of contributions
from all other persons.

3
4
5
6
1
2
7
8
Applying Katz's Path-counting algorithm
Longest path:
1 4 5 8 3 6 7
use b = 0.25
0.0
0.25
1.30
0.25
0.56
0.58
1.98
0.70
s5
= b (1 5) + b (4 5)
+ b2
(1 4 5)
s8
= b (4 8) + b (5 8)
+ b2
(1 4 8)
+ b2
(1 5 8)
+ b2
(4 5 8)
+ b3
(1 4 5 8)

Features of Katz's Path-Counting algorithm
● More than just direct link counting
○ Indirect links are taken into account
● The constant b is an effectiveness of a link. It is
accumulated along a path.
● Problems when there are cross links or cycles in the graph
○ Cross links: every non-zero with the path going through one
node in a cross link implies nonzero , generally an over-
estimate.
○ Not an issue with journal publications.
○ Web pages can be updated, making cycles a common
phenomena.

The HITS algorithm by Kleinberg (1999)
● HITS = Hyperlink-Induced Topic Search, a.k.a. “Hubs and
Authorities” algorithm.
● From a pre-selected graph of N pages, try to find hubs
(out-link dominant) and authorities (in-link dominant).
○ assign page k a hub score hk
and an authority score ak
.
○ aj
is the sum of hi
over all pages i that has (i j) link.
hi
is the sum of aj
over all pages j that has (i→j) link.
where A is the adjacency matrix.
○ normalized vector h = (h1
, h2
, … hN
), a = (a1
, a2
, … aN
)

HITS algorithm (cont.)
● Start with a0
= h0
= (1, 1, … 1), iterate until converge.
○ convergence to the vector with largest | | is guaranteed by
eigenvalue theory.
● Pick top X pages with highest authority scores to be the
best search results.

3
4
5
6
1
2
7
8
Applying HITS algorithm
0.00
0.41
0.14
0.41
0.57
0.24
0.14
0.64
0.36
0.33
0.08
0.21
0.63
0.00
0.33
0.19
Authority scores
Hub scores

Pinski-Narin: Influence (1976)
● “Not all citations are created equal”
● Impact of journal j = influence weight of journal j = rj
● Wij
= fraction of citations of journal i that goes to journal j
● Influence of journal j = weighted sums of influences of
journals that cite journal j
○ More citations gives higher influences, citations from
influential journals gives higher influences.
● r is the eigenvector of matrix WT
with eigenvalue 1.
○ Question: Does such a solution always exist?

Calculation of influence
● From the eigenvalue equation
we can use the Power Method to find the eigenvector of the
largest eigenvalue.
● Start with r0
= (1, 1, … 1)T
.
● Calculate rk
= WT
rk-1
.
● Repeat until convergence is reached:
● Questions:
○ choice of r0
?
○ convergence?

Applying Pinski and Narin's influence
3
4
5
6
1
2
7
8
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
?

Eliminate both the source and the sink
3
4
5
6
1
2
7
8
0.07
0.02
0.27
0.02
0.02
0.14
0.23
0.24

Features of Pinski and Narin's Influence
● Propagation of influences
● Geller introduced Markov chain interpretation to
in 1978.
○ W is a stochastic matrix.
○ Markov process: reader randomly start with a journal, then
choose another journal randomly from the citations.
○ rj
= the stationary probability that journal j is read. It's natural
to use the L1 norm for the r vector, i.e.,
○ Perron–Frobenius theorem: principal eigenvalue = 1 for
stochastic matrices.

What's wrong?
The sink
The source
It's no longer a transition matrix!
No solution is guaranteed if there are sinks or sources. A
group of nodes linking to each other but nowhere else is

The PageRank algorithm
● Improve over Pinski-Narin's fragility on sinks by introducing
a random surfer model.
○ A random surfer reads pages, randomly following links or
jump to a random page:
where v is a “preference vector” as a probability distribution:
○ rj
= the probability that page j is read.
○ For the case of vj
= constant, it becomes
● Use power method to solve r.

Applying PageRank algorithm
3
4
5
6
1
2
7
8
1.9
2.3
6.7
2.3
2.3
4.7
11.4
3.5
PageRank * 100
use =
0.85

Without sinks and sources
3
4
5
6
1
2
7
8
use =
0.85
7.3
3.4
25.3
3.4
4.2
12.7
22.0
23.1
PageRank * 100

The rank sink problem
● The sink is still a problem, although convergence is
reached and non-zero values are obtained. Why?
● Re-write the PageRank equation to eigenvalue form:
where
● Is P a column-stochastic matrix?
● Wait... first term = 0 if node i is a sink!


The Google matrix
● Define
where aj
= 1/N if node j is a sink node, 0 otherwise.
● Now G is a column-stochastic matrix.
e is a column vector with all 1's.
● The final PageRank equation is
○ PageRank Pinski-Narin influence as α 1 .
○ PageRank uniform constant as α 0.

PageRank from the Google matrix
3
4
5
6
1
2
7
8
5.3
6.5
19.2
6.5
7.8
13.5
32.5
10.1
PageRank * 100
use =
0.85,
v = e / N

Discussion on PageRank
● Is PageRank guaranteed to converge?
● What's the reasonable value of α? How sensitive is the
ranks with respect to it?
● How to calculate it for the web graph?
● Independent of query – need to add query-dependent
rankings for real world applications.

Convergence of PageRank Calculation
● The speed of convergence of the power method is
○ It can be proven that for the Google matrix.
○ Brin and Page used 50 to 100 iterations to converge,
0.8550
= 3 x 10-4
.
● Choosing close to 1 emphasizes the link structure of the
web, but the convergence will be much slower.
● Choosing close to 0 converges quickly, but the result is
close to uniform ranks and useless.

Stability of PageRank
● Evaluate the change of principal eigenvector when the
Google matrix is perturbed by changing α or the link
structure.
● The condition number of G is shown by Kamvar and
Haveliwala to be
○ Let
○ Then
○ Implications:
■ PageRank is not stable for close to 1.
■ PageRank is reasonably stable for < 0.9.

More meanings of the α parameter
● Revisit the PageRank equation:
● With uniform v, it becomes
Compare with Katz's path-counting algorithm:
The α constant plays a role of damping of influence along a
path. That's why it is called “damping factor” in later
literatures.
○ S vs. A: S has branching factor 1/L, a penalty for cycles.

Calculating PageRank for the web
● Again, web is huge.

Practical considerations
● Storage:
○ The G matrix is dense due to the sink nodes – make use of
the sparse W matrix and rank-one updates aj
and vj
.
○ Store Wij
, aj
, vj
and rj
(k)
: O(10N) + O(N) + 2N = O(N) ~ O(TB)
● Calculation:
○ The power method can easily be parallelized (it better is!)
○ Decomposition of G matrix is one way
○ Rank forwarding is another
○ Use your creativity!

The matrix decomposition method
+
Do you see the problem of this approach?

Rank forwarding approach
● Use the equation
● Phase 1: outgoing links
○ A master process reads data, and send data of one node to
one process.
input = i, out-links j1
, j2
, ..., jL
, vi
, PageRank ri
(k)
output = key-value pairs (j1
, ri
(k)
/L), … (jL
, ri
(k)
/L),
(i, ai
+ (1- )vi
) {key = node id}
● Phase 2: shuffle:
○ Send (key, value) pairs of the same key to the same process
● Phase 3: incoming links
○ Each process calculates one node at a time
input: (j, v1
), (j, v2
), …
output = (j, Σ v) = (j, rj
(k)
), and write to disk

MapReduce
● The 3-phase calculation is applicable to various massive
data processing tasks
○ Read a lot of data
○ Map: extract something you care about from each record
○ Shuffle and Sort
○ Reduce: aggregate, summarize, filter, or transform
○ Write the results
● Google developed a framework around this paradigm of
parallel computing, called MapReduce.
○ Data analyst write a mapper function and a reducer function
○ Specify the input data, and the framework takes care of the
rest

Parallel MapReduce
Map Map Map Map
Input
data
Reduce
Shuffle
Reduce
Shuffle
Reduce
Shuffle
Partitioned
output
Master

MapReduce scheduling
● One master, many workers
○ Input data split into M map tasks (typically 64 MB in size)
○ Reduce phase partitioned into R reduce tasks
○ Tasks are assigned to workers dynamically
○ Often: M=200,000; R=4,000; workers=2,000
● Master assigns each map task to a free worker
○ Considers locality of data to worker when assigning task
○ Worker reads task input (often from local disk!) to produce R
local files containing intermediate k/v pairs
● Master assigns each reduce task to a free worker
○ Worker reads intermediate k/v pairs from map workers
○ Worker sorts & applies user’s Reduce function to produce the
output

Some features of MapReduce
● Backup tasks can be spawned near end of completion –
don't get stuck with hang tasks.
● Mapper often reads from local disk – schedule process to
data, reduce network load.
● Robust fault tolerance:
○ Detect worker failure via periodic heartbeats
○ Re-execute map tasks and/or reduce tasks as necessary
○ Master state is checkpointed to Google File System: new
master recovers & continues
○ Lost 1600 of 1800 machines once, but finished fine.

Some published usage statistics
Number of jobs
Aug, ‘04
29,423
Mar, ‘05
72,229
Mar, ‘06
171,834
Average completion time (secs) 634 934 874
Machine years used 217 981 2,002
Input data read (TB) 3,288 12,571 52,254
Intermediate data (TB) 758 2,756 6,743
Output data written (TB) 193 941 2,970
Average worker machines 157 232 268
Average worker deaths per job 1.2 1.9 5.0
Average map tasks per job 3,351 3,097 3,836
Average reduce tasks per job 55 144 147
Unique map/reduce combinations 426 411 2345

IF vs. PR by Bollen et. al. (2006)
● Formed a journal citation graph with 2003 ISI Journal
Citation Reports data set. 5710 journals have non-zero
citations from 2002 and 2001.
● Calculated Impact Factor and PageRank, and multiply
them as the “Y-factor”.

IF vs. PR in Physics
Does Impact Factor result make sense to you?

Issues of citation counting
● Some of the most cited journals are review articles or data
tables. They are popular and of utility values, but counting
such citations toward impact doesn't make much sense.
● Journal editors/referees may encourage authors to cite
their journals to increase impact factor (Smith 1997).
○ Spamming the citation count “at no cost”.
● Length bias: “communications” and “letters” type of journals
may cite less due to length limit.
● Many other arguments can be found on the literature.

Conclusions
● PageRank goes beyond link counting and forwards
standing/influence/impact of each node to the nodes it links
to.
● PageRank vector as the stationary probability vector of the
Markov chain specified by the Google matrix has many
nice properties and has been extensively studied.
● It has been applied to journal status in literatures, but didn't
catch on – probably the calculation is not simple enough
compared to impact factor and h-index.
○ But it is actually very easy to implement for small networks!
● Maybe you can give it a try on your network

References
● Page, Brin, Motwani, and Winograd, “The PageRank
Citation Ranking: Bringing Order to the Web,” Technical
Report. Stanford InfoLab (1998).
● L. Katz, “A new status index derived from sociometric
analysis,” Psychometrika, 18(1953), pp. 39–43.
● E. Garfield, “Citation analysis as a tool in journal
evaluation,” Science, 178(1972), pp. 471–479.
● G. Pinski, F. Narin, “Citation influence for journal
aggregates of scientific publications: Theory, with
application to the literature of physics,” Inf. Proc. and
Management, 12(1976), pp. 297–312.
● N. Geller, “On the citation influence methodology of Pinski
and Narin,” Inf. Proc. and Management, 14(1978), pp. 93–
95.

References (II)
● Amy N. Langville and Carl D. Meyer, “Google's PageRank
and Beyond: The Science of Search Engine Rankings”,
Princeton Press (2006).
● Jon Kleinberg, “Authoritative Sources in a Hyperlinked
Environment,” J. ACM Vol. 46, 5 (1999): 604–632.
● Ng, Zheng and Jordan, “Link analysis, eigenvectors and
stability”, IJCAI 17, 1: 903-910 (2001).
● Barabási and Albert, "Emergence of scaling in random
networks", Science, 286:509-512, October 15, 1999.
● Albert, Jeong and Barabási, "The Diameter of the WWW”,
Nature 401: 130-131 (1999).
● Bollen, Rodriquez and Van de Sompel, “Journal status”,
Scientometrics 69, 3: 669-687 (2006).

References (III)
● Seglen, “Why the impact factor of journals should not be
used for evaluating research”, BMJ 1997;314:497.
● Smith, R., “Journal accused of manipulating impact factor”,
BMJ 1997;314:7079.

Page rank talk at NTU-EE

More Related Content

Viewers also liked (6)

Similar to Page rank talk at NTU-EE (20)

Recently uploaded (20)

Page rank talk at NTU-EE