Impact Factor and the Evaluation of Scientists - a book chapter by Nicola de Bellis

Presentation by Xanat V. Meza
for the New Media class

 Scientific progress rests on the recognition
and legitimization of individual contributions
by a research community sharing notions,
methodologies, practices and values.
 Scientific quality: the assessment of the
novelty of a contribution through the
evaluation of scientific publications.

 Peer reviewing: examination of scientific
publication’s content by a group of
acknowledged experts.
 Born in XVIIth century Europe against
knowledge fragmentation caused by scientific
specialties.
 The evaluation should be:
◦ Encoded.
◦ Impersonal.
◦ Convergent of judgment criteria from different
experts.

 Kuhn: there is a resistance to change, but
conservatism is also the staple of scientific
change. Pg. 182
 From 1970’s the use of citation analysis
attracted the attention of politicians and
science managers, as it is convenient, quickly
understood, easily applied, and easy to
calculate thanks to ISI databases.
 The problem is with unpublished material,
which has to rely on qualitative judgment of a
local character.

 Scientometricians have adopted a dual
strategy to make sure that quality equals
citation impact:
◦ 1. Why it is supposed to work: statistical evidence
points to the fact that excellent scientists often
publish citation classics.
◦ 2. How to avoid rejections: by anticipating the
ambiguous role of self-citations and delayed
recognition.

 On self-citation: Tagliacozzo (1997). They are
suspected of deceitfully inflating the citation
impact of the unit under assessment.
Nevertheless, they can be used as an impact-
reinforcing mechanism by triggering a chain
reaction.
 Another problem might be the historical
recurrence of premature discoveries (pg.
185).

 The Impact Factor (IF) is a journal citation
measure devised in the early 60’s by Garfield
and Sher for Current Contents and the SCI.
 It is an estimation of a journal’s average
article’s citation score over a relatively short
time span.
 It is computed for a given year through a
division between a numerator and a
denominator.

 The numerator is the number of citations
received in the processing year by the items
published in that journal during the previous
two years.
 The denominator is the overall number of
citable items (research articles, reviews, and
notes) issued by the journal during the same
two years.
 X is the name of the journal.
IF(X) = 100 + 150 = 3.57
70

 The proportional increase in the IF score became
a prelude to marketing success, to an increased
commercial and symbolic visibility, and
occasionally to a more profitable sale of
advertising spaces.
 Arguments against the IF:
◦ Skewness of citation distribution: the poor statistical
correlation between the citedness of individual articles
and the IF of the journals wherein they appeared is
well documented.
◦ The several conceptual and technical limitations
bearing upon the significance of the IF.

 Concerns about the stability and reliability of
citation rankings were expressed in 1970
within the American and the European
(specifically Dutch) bibliometrics subculture.
 Narin’s Evaluative Bibliometrics and Dennis
Dieks and Hans Chang’s 1976 paper (pg. 189).
 Nancy Geller estimated the lifetime citation rate
of a paper under a series of suppositions about
the regularity of citation patterns and the
growth rate of scientific literature.

 Allison attempted to provide a scale-
invariant measure of inequality in
publication/citation counts in 1980.
 Schubert and Glanzel designed a reliability
test for determining the statistical
significance of the observed differences in
citation impact in 1983.
 Bensman performed an exploratory
investigation of the 2005 JCR’s probabilistic
structure (pg. 190).

 The confirmations are:
◦ Total citation count and IF capture different
facets of journal importance,
◦ The former is better than the latter as a global
measure of importance, but the gap narrows if
only a better classification is introduced in the
sample sorting journals from research journals.
◦ Both measures are surprisingly stable over time
at the higher level of citation rankings.

 The objections to the IF:
◦ Classification of citable items: the number of citable
items does not take into account letters, editorials or
conference abstracts.
◦ Accuracy issues: journals are complex entities that
can change, split, merge, etc. The ISI doesn’t combine
citation data on the basis of lineage, nor for sections
of the same journal (pg. 192).
◦ Density and age of cited references. The more one
cites, the more can be cited. Density and age of cited
references emphasize the variability of citation
cultures among disciplines and research fields.
◦ The Journal format and article type: the speed and
intensity with which different types of articles attract
citations affects the IF.

 Modification of the time window for either the
cited or the citing years.
 Creation of a normalized measure, taking into
account the (sub)field citation practices, types
of documents published by the journal, and
age of cited papers:
◦ Graeme Hirst measured the number of times a journal
is cited by the core literature of a single subfield in
the 1970’s.
◦ Pinski and Narin made a Google-like algorithm for
journal ranking.
◦ The Journal to Field Impact Score introduced by van
Leeuwen and Moed.

 It counts the same items than the IF both at the
numerator and denominator.
 It is field-specific, in the sense that the impact
of the individual journal is compared to the
world citation average in the corresponding
research fields.
 It differentiates the normalized impact for the
various document types.
 It employs variable citation and publication
windows for the count depending on the
communicative patterns of the research field
under evaluation.

 In order to improve the ISI, Leiden
bibliometricians at the CWTS completed a pilot
study in 2006. They expanded the ISI indexes
with source papers from refereed proceedings
of computer science international conferences
in view of developing field-specific bibliometric
indicators.
 Grant Lewison envisioned the expansion of
analytical tools necessary to trace the routes
along which biomedical research influences
health decisions. This includes patents, clinical
guidelines and newspapers.

 The opponents of quantitative methods get rid
of bibliometric indicators because nothing
appears as reliable as an accurate peer review.
 But some place so much trust in quantitative
analysis that claim that properly weighted
indicators should be implemented by expert
systems and computer-assisted procedures to
help determine career progression and
university chair assignment.
 Bibliometricians recognize the importance of
peer review and the implementation of
additional, less subjective analytical tools.

 Rarity is a structural property of the citation
network.
 After the launching of the SCI, statistical
surveys revealed that the ratio between
references processed each year and the
number of unique items cited by those
references was nearly constant and
approximately equal to 1.7 (the Garfield’s
constant).
 It means that in a single year, each paper was
cited on average only 1.7 times and 25% of the
papers were never cited.

 Normalization is usually attained by relating
the citedness of a set of papers to a
conventional standard that may be either
relative or absolute:
 A relative standard is the citation score of a
“control group” of papers allegedly similar to
those under evaluation. Co-citation analysis or
bibliographic coupling can be used.
 An absolute standard is the expected number
of citations per paper in the research (sub)field
encompassing the papers under scrutiny.

 Schubert and Braun introduced a relative
citation rate indicator for papers published
in the same journal that relates the number
of citations actually settled on them to the
mean citation rate of all papers appearing
in that journal.
 H-index was proposed by Jorge Hirsch in
2005.
 It means to provide a joint characterization
of both productivity and cumulative
research impact.

 A scientist has index h if h of the papers he or
she has (co)authored have at least h citations
each, while the rest have fewer than h citations
each.
 The subset of medium-highly cited papers
bearing on the calculation of the h has been
dubbed “h core” by Rousseau.
 The author of many low-cited papers will get
as weak an h-index as the one who publishes
only a few “blockbusters”.
 The Journal of Informetrics devoted a special
issue to h-type indexes in 2007.

 But!
◦ H-values cannot exceed the number of a
scientist’s publications and don’t decrease for
those who give up publishing or don’t get
citations from a certain point on.
◦ That’s why Hirch’s seminal paper also suggested
dividing h by the years of academic activity.
◦ It overlooks publication type and age, citation
age, self-citation rate, and number of coauthors.
 Variations and corrections to the h-index:
pg. 204.

 Bibliometricians believe that citation analysis
applied to the corpus of publications produced
over a certain period of time by the members of a
collective entity deals with a number of items large
enough to allow a fairly safe application of
standard statistical tools.

Impact Factor and the Evaluation of Scientists - a book chapter by Nicola de Bellis

 Issues: comparing output and impact data of
research organizations with sharply dissimilar
organizational profiles, missions, managerial
culture, financial resources, and research
facilities.
 Cross-country and cross-field comparison.
 Systematic errors:
◦ Limitation of citation indexing to the first author in
the case of multiauthored papers
◦ The decision to not provide unified citation counts for
journals undergoing complex editorial changes
◦ The criteria applied to the selection of source journals

 They also don’t pay too much attention to non-
Anglo-American journals, non-English-
language journals, and nonjournal materials.
 Since 1907 local databases and in-house
software for storing and processing ISI citation
data has been created on these regards:
◦ National Science Foundation’s Science Literature
Indicators Database
◦ ISSRU at the Library of the Hungarian Academy of
Sciences
◦ CWTS at Leiden University.

 The most authoritative research on how
field-specific and reliable bibliometric
measures ought to be defined:
 “Googling” citation networks: Pinski and
Narin. The influence methodology
introduced a journal ranking algorithm
inspired by the basic principle of social
networking: citations are not all equal, their
weight being adjustable as a function of the
prestige of the citers.

 The influence weight is a size-dependent
measure of the weighted number of
citations a journal receives normalized by
the number of references it gives to other
journals.
 The influence per publication for a journal
is the weighted number of citations each of
its articles receives from other journals.
 Google’s pagerank, Eigenfactor algorithm,
recently proposed variants of the IF.

 Big science bibliometrics: Martin and Irvine. The
methodology of converging partial indicators
appeals to combine several bibliometric and non-
bibliometric indicators, including publication
counts, citation analysis, and an extensive form of
peer review fed by direct interviews with scientists.
It is relative and comparative.
 The Hungarian way. Scientometricians devised a set
of relative indicators of publication output and
citation impact that allow cross-field comparisons
among countries, research institutes, departments
and scientific societies in a mathematical fashion.

 The Hungarian way. Indicators for countries:
 Activity Index (AI): it is the ratio between the
country’s share in the world’s publication
output in the field and the country’s share in
the world’s publication output in all science
fields combined.
 Attractivity Index (AAI): it is the ratio between
the country’s share in citations attracted y
publications in the field and the country’s share
in citations attracted by publications in all
science fields combined.

 The Hungarian way. Indicators for
countries:
 Activity Relative Citation Rate (RCR): the
ratio between a summation of observed
values and a summation of expected values
for all the papers published by a country in
a given research field:
RCR = Sum Observed citation rate
Sum Expected Citation rate

 The Leiden School. Their methodology
disregards the analysis at the macro-level
of the country, charged with being too
generic to characterize research
performance in a politically relevant
fashion, and traces the roots of scientific
excellence to the university and its
operative units.
 The hallmark of scientific interest is based
on publishing and highly cited papers.

 The world average is the ratio between the
average number of citations per publication
(corrected for self-citations) and a field-
specific world average based on the citation
rate of all papers appearing in all journals
belonging to the same field in which the
unit under evaluation has been publishing.
 After a series of papers by Martin and Irvine
on the 1980’s regarding the decline of
British science, other authors resorted to
alternative versions of ISI databases.

 Leydesdorff found a relative stability followed
by a remarkable increase on British science.
 Braun, Glanzel and Schubert argued that there
were only random fluctuations.
 The issue was on how to handle the raw data:
◦ Fixed journal set or dynamic set use.
◦ Computing annual publication totals on the basis of
tape-years, the date a publication entered the SCI, or
the publishing date.
◦ Limiting countable output to specific publication
types.
◦ Adopting a fractional author count in the case of
multiauthored papers.

 The Internet and the World Wide Web
introduction were sign of the emergence of a
global and “knowledge-based” economy.
 Business competition is exercised through the
control of natural resources, commodity
markets, low-cost manpower and deployment
of investable intellectual capital.
 Knowledge-drive innovation is now integral to
commercial success
 Products stimulate innovation but inhibit
diffusion through intellectual property
restrictions: patents.

 Patent: legal document issued by the
government. In exchange for the public
disclosure of the technical details of an
invention, grants the inventor or any person or
organization to whom the inventor’s
prerogatives have been transferred, the
monopoly on its production and commercial
exploitation.
 Since most inventions are built upon previous
objects or techniques, the verification of
patentability requires an in-depth analysis of
the invention’s technical specifications by a
skilled examiner.

 A typical U.S. patent is composed of:
◦ A title page with bibliographic data and practical
information to identify the document unambiguously.
◦ The description of the invention explaining how to
make and use it.
◦ The claims defining the scope or boundaries of the
patent.
 Only a small fraction of research output is
patented.
 An invention should be novel, nontrivial and
commercially exploitable.

 Patents are though to manage mainly because of
the extent of their content’s dependence on
scientific knowledge, bearing on the basic issue of
the relationships between technology and science.
 We could reduce technology to applied science, but
it is not that easy.
 Bibliometrics is asked to provide factual evidence
to extend to technological documents the same
analytical techniques applied to scientific literature,
both for quality assessment purposes and for
mapping the formal connections between scientific
and technological research areas.

 The Gross Domestic Product simply counts
and classifies patents, but don’t tell us the
weight of each patent’s contribution to
economic and technological advancement.
 Using citations as an aid to effective patent
searches alternative (or complementary) to
subject-based classification codes was
circulating among American patent
attorneys since the 1940’s.

 In 1957, Garfield tested a patent citation index
to 4 000 chemical patents.
 The official version was published in the 1964
and 1965 editions of the SCI, including all U.S.
patents.
 It was dropped due to lack of financial support.
 Reisner tested a machine-readable citation
index to patents as a tool for monitoring the
performance of classification systems.
 It was found that if many patents were build
upon an specific citation, this citation was a
significant “technological spillover”.

 The interest in patent citation analysis has
flourished since the 1980’s, when large-scale
computarized patent data became increasingly
available for automatic processing.
 Narin’s team extended the core of bibliometric
techniques to technology indicator
construction.
 Jaffe and Trajtenberg employed patent citations
to quantify the market value of patents and the
flows of technological knowledge in the heart
of economic growth.

 There are a wide range of indicators of
technological prominence and diffusion
under design:
◦ Knowledge diffusion.
◦ Technology and science.
◦ Evaluation studies: Narin.
◦ Business intelligence: Narin.
 In high-tech and fast-moving areas, there
is a striking similarity between the
referencing cycles of cientific articles and
those of patents.

 Patents have been found emmeshed in
sale-free citation networks governed by a
power law distribution that imposes an
uneven allocation of symbolic wealth
among units of supposedly different caliber.
 Patent references are the result of a social
process involving at least three actors: the
inventor, the attorney or agent, and the
patent examiner.

Impact Factor and the Evaluation of Scientists - a book chapter by Nicola de Bellis

More Related Content

What's hot (20)

Viewers also liked (10)

Similar to Impact Factor and the Evaluation of Scientists - a book chapter by Nicola de Bellis (20)

Recently uploaded (20)

Impact Factor and the Evaluation of Scientists - a book chapter by Nicola de Bellis

Editor's Notes