Novelty and Impact of Economics Papers

To test our hypothesis that different forms of novelty predict different types of scientific influence, we must first define impact in a multidimensional manner. A paper might attract widespread attention, reflected in high citation counts, while another might receive modest immediate attention yet fundamentally redirect its field’s trajectory. To capture these distinct pathways, we employ two complementary metrics that capture distinct dimensions of scientific impact: citation count and the disruption index.

The study of scientific progress hinges on the concept of novelty, yet its measurement remains a central challenge. Prevailing methods, such as the widely-used atypicality score that measures unconventional reference combinations (Uzzi et al., 2013), have provided powerful insights. However, these approaches share a common conceptual foundation: they treat novelty as an intrinsic attribute of a paper, inferred from its inputs (e.g., references) or its internal content (e.g., new keywords). This view, while intuitive, limits our ability to understand the strategic and ecological dynamics of scientific discovery.

We propose a fundamental shift in perspective, reconceptualizing novelty not as an intrinsic property but as a relational construct. In our view, a paper’s novelty is a function of its location within a high-dimensional semantic space, characterized by both its static isolation and its dynamic evolution.

To characterize a paper’s position, we decompose it into two orthogonal dimensions. These dimensions describe its relationship with the dynamic “semantic neighborhood” it inhabits:

• •

  Spatial novelty measures a paper’s static position in the semantic space. It quantifies intellectual distinctiveness by measuring a paper’s isolation from the accumulated stock of prior knowledge.

• •

  Temporal novelty measures the dynamics of its position. It captures strategic timing by measuring the change in a paper’s immediate semantic over the period leading up to publication, reflecting its engagement with an evolving research frontier.

These two dimensions are central to our analysis, as we hypothesize they correspond to distinct strategic positions with fundamentally different consequences for scientific impact. To operationalize this framework, we develop a family of semantic isolation metrics, which we describe next. These metrics are designed to quantify a paper’s static isolation (for spatial novelty) and the change in its isolation over time (for temporal novelty) using text embeddings.

We construct semantic isolation metrics to quantify both spatial and temporal novelty. Each metric is based on the semantic distance between papers, computed by embedding every paper’s full text into a high-dimensional vector space (see Section 3.2) and measuring pairwise distances. For spatial novelty, we define four point-in-time metrics, each capturing a distinct facet of a paper’s semantic distinctiveness relative to the existing literature. For temporal novelty, we derive dynamic metrics by tracking how each point-in-time measure evolves across successive time windows around a paper’s publication (e.g., three years pre-publication versus publication). The resulting slopes serve as our indicators of well-timed engagement with emerging research frontiers. We synthesize these individual measures into two composite scores, as detailed in Section 5.1.

In this section, we provide a conceptual illustration of semantic isolation metrics. A full technical description of these metrics appears in Appendix A.

Our analysis draws on a comprehensive corpus of scholarly articles in economics. The initial dataset comprises 105,082 articles published between 1980 and 2020, sourced from RePEc, Google Scholar, and SciSciNet (Lin et al., 2023). From SciSciNet, we obtain key bibliometric variables, including the disruption index and the atypicality score. We supplement this with citation counts collected from Google Scholar in December 2024.

To ensure a robust measurement of novelty and impact, we partition our sample by time. Papers published between 1980 and 1995 form a ”knowledge base,” providing the necessary historical context to measure the novelty of subsequent work. Our analysis focuses on a ”focal period” cohort of papers published between 1996 and 2015. This design allows us to observe impact outcomes over a meaningful time horizon (at least 9 years post-publication) while grounding our novelty measures in a deep historical context. After filtering for data quality and completeness, our final analysis dataset contains 52,766 papers.

Table 1 provides descriptive statistics for the focal sample, disaggregated by subfield (classified using a large language model). The data reveal substantial heterogeneity across fields in both citation patterns and the prevalence of disruptive work. For instance, Development and Agricultural Economics have the highest median citation counts and disruptive rates, respectively, while Microeconomic Theory exhibits lower values on both dimensions. As expected, the distribution of citations is highly right-skewed across all fields.

Our primary data consist of full-text research articles in PDF format. We developed a multi-stage pipeline to convert these documents into structured semantic representations suitable for calculating our isolation metrics.

First, we convert the PDFs to plain text using an LLM-based Optical Character Recognition (OCR) model. Second, we use LLMs to generate comprehensive summaries of each paper. A key innovation of our approach is the extraction of multiple, structured representations of each paper’s intellectual content. Rather than relying on a single summary, we prompt the LLM to decompose each paper into five distinct semantic dimensions:

• •

  Paragraph: A concise, high-level overview of the paper’s content.

• •

  Topic: The paper’s subfield, primary subject matter, and specific issues addressed.

• •

  Question-Conclusion: The core research questions and key findings.

• •

  Methodology: Data sources, analytical models, and technical approaches employed.

• •

  Theory: The theoretical frameworks, conceptual models, and analytical innovations.

This decomposition allows us to measure novelty along different dimensions of a paper’s contribution - for example, distinguishing methodological novelty from theoretical novelty.

Third, to ensure the robustness of our semantic representations, we embed each of the five summary types using three complementary embedding models: SciBERT (Beltagy et al., 2019), a BERT model pre-trained on a large corpus of scientific literature; SPECTER (Cohan et al., 2020), a model specifically designed to generate document-level embeddings for scientific papers; and Qwen (Li et al., 2023), a general-purpose state-of-the-art LLM. This triangulation approach mitigates the risk of model-specific biases.

Finally, we calculate pairwise distances between all paper embeddings using both Euclidean distance and cosine similarity. This yields a comprehensive feature space derived from the combination of five summary types, three embedding models, and two distance metrics. This systematic process results in 3240 candidate isolation features for each paper, providing a rich basis for our analysis. Further details are provided in Appendix B.

We evaluate the predictive power of our isolation metrics through two supervised learning tasks: predicting long-term citation counts and identifying disruptive papers. Our experimental design is structured to test two key hypotheses: (1) that our isolation metrics have significant predictive power in isolation and (2) that their predictive content is not redundant with other textual or bibliometric features. This validation strategy rests on a key premise: if our semantic isolation metrics capture meaningful dimensions of novelty, they should predict future impact beyond what is explained by existing measures and standard bibliometric controls. Strong predictive performance, particularly when competing against established novelty proxies like atypicality, provides evidence that our metrics identify papers positioned for significant influence.

We define several feature groups for our predictive models:

• •

  Controls: Basic metadata including publication year, reference count, document length, word count, and sentence count.

• •

  Subfield (Baseline): Indicator variables for the LLM-classified subfields.

• •

  Journal: Journal-specific features including encoded journal identifiers and top-5 journal indicators.

• •

  Textual Features: Standard NLP features including word counts, part-of-speech tag frequencies, and other text-derived metrics.

• •

  Topic Models: Latent Dirichlet Allocation (LDA) and Term Frequency–Inverse Document Frequency (TF-IDF) representations. They contains future information that is not available at the time a paper is published, as they are calculated with the whole dataset.

• •

  Atypicality: The combination-based novelty scores developed by Uzzi et al. (2013), including 10th percentile, median, and number of combinations, as provided by SciSciNet (Lin et al., 2023).

• •

  Isolation: Our proposed set of semantic isolation metrics, including point-in-time isolation metrics and retrospective dynamic isolation metrics²²2We do not include perspective dynamic isolation metrics in our main analysis. The version with perspective dynamic isolation metrics is in Appendix C.4..

To fully capture the nonlinear relationship between features, we employ XGBoost for prediction tasks. We use consistent hyperparameters across all feature combinations to ensure performance differences reflect information content rather than optimization artifacts. These hyperparameters are identified by tuning the model on the most complex feature set (the union of all feature groups), as parameters that perform well on this high-dimensional space are the most likely to be robust and generalize effectively to simpler feature subsets. All results are averaged over 100 bootstrap iterations for robust statistical inference.

We first assess the ability of our isolation metrics to predict long-term citation counts, a standard proxy for scientific impact. Our primary evaluation metric is the out-of-sample Mean Squared Error (MSE). We also report the ”completeness score” from Fudenberg et al. (2022), which measures the proportional reduction in MSE relative to a baseline feature set (Controls + Subfield). The larger the completeness (maximum is 1), the more error the tested model reduce from the base, which means the better the model’s performance. For example, using all features together gives a completeness score of 34.92%, meaning that this large set of features can reduce 34.92% of our baseline prediction error. This suggests that predicting log citation counts is a hard task.

We next turn to a more nuanced prediction task: identifying disruptive papers. This allows us to test whether our isolation metrics are merely a proxy for future attention (citations) or if they capture a more fundamental aspect of a paper’s contribution. We frame this as a binary classification task, classifying a paper as disruptive if its Disruption Index (DI) is greater than zero (Wu et al., 2019).

Given the class imbalance (16.3% of papers are disruptive), we focus on precision as our primary evaluation metric. Our goal is showing that semantic isolation is a high-quality signal that reliably predicting disruptive papers. Precision answers the question: “When our model predicts a paper is disruptive, how often is it correct?” A high precision score provides strong evidence that semantic isolation is a reliable signal of disruptive potential. On the other hand, recall, which measures ability to identify every disruptive paper, is of secondary importance. We also report the F1-score and the completeness score (where error is defined as ($1-precision$). We use balanced class weights during training and report mean F1 and precision across 100 times bootstrapping.

Our predictive analyses provide robust evidence for the empirical relevance between our semantic isolation metrics and future impacts. Across two distinct impact measures - citations and disruption - these metrics demonstrate substantial predictive power, significantly outperforming the established atypicality benchmark. Crucially, their predictive content is both complementary to atypicality and non-redundant with a comprehensive set of textual and bibliometric controls, confirming that they capture a novel dimension of scientific contribution rooted in a paper’s semantic positioning and temporal dynamics. Notably, these features are calculated with information available at the publication time of a paper, making it a useful tool to assess the future impact at a paper’s early stage.

As detailed in Appendix C.3, these findings are robust across economic subfields. Decomposition (see Appendix C.2) reveals that features related to papers’ core contributions (paragraph summaries and question-conclusion framing) provide the strongest predictive signals. Appendix C.4 shows that including prospective metrics can significantly improve the predicting performance, serving as a validation of our temporal novelty concept.

Having established the empirical relevance between our semantic isolation metrics and impacts, we now proceed to use them as the measurement foundation for our two-dimensional framework of novelty.

We operationalize our framework by constructing two composite scores that correspond to the conceptual dimensions of novelty outlined previously. These scores are calculated using only information available at the time of publication, ensuring clear interpretation of temporal relationships.

While we have a large set of semantic isolation features, we need to derive scalar values for spatial novelty and temporal novelty. Simple aggregation methods such as taking the mean are straightforward but would lose the nuances captured by individual features. For example, averaging the 30-NN distance and 3-NN distance would either be meaningless or dominated by the larger 30-NN values. Therefore, we employ Principal Component Analysis (PCA) to preserve most of the variation in the features while reducing their dimensionality.

• •

  Spatial Novelty Score: We apply PCA to the set of contemporaneous point-in-time isolation metrics (i.e., metrics calculated using all prior work, $t=0$) and the baseline paper-level mean distance⁵⁵5Basic features of semantic distance, introduced in Appendix A.4.. The first principal component serves as our spatial novelty score. This score captures a paper’s static semantic isolation from the existing body of knowledge at its moment of publication.

• •

  Temporal Novelty Score: We apply PCA to the set of retrospective dynamic isolation metrics, which measure the change in a paper’s semantic neighborhood in the years leading up to its publication (e.g., the change from $t=-3$ to $t=0$). The first principal component serves as our temporal novelty score, capturing the degree to which a paper is situated within a rapidly evolving or ”hot” research frontier.

The use of contemporaneous point-in-time metrics and retrospective dynamic metrics makes sure that our framework are not using any future information. By using the first principal component for each, we create summary indices that capture the largest shared dimension of variation across our granular metrics, reducing measurement noise while retaining the core signal of each novelty type. Further details on the PCA construction are provided in Appendix D.1.

We classify each paper based on the state of its semantic neighborhood, defined by its position in the two-dimensional novelty space. A neighborhood is classified as ”High” on a given dimension if the paper’s score on that dimension is above the median. Crossing these two classifications yields a four-quadrant typology, with each quadrant representing a distinct archetype of research environment:

• •

  Consolidating Neighborhoods (Low Spatial / Low Temporal): These are stable, well-established research areas where contributions are semantically close to existing work. Papers in these neighborhoods typically refine or incrementally extend established paradigms.

• •

  Outlying Neighborhoods (High Spatial / Low Temporal): These are intellectually distant and slow-moving or static research areas. Papers here represent unconventional ideas disconnected from contemporary research trends.

• •

  Trendy Neighborhoods (Low Spatial / High Temporal): These are rapidly evolving research frontiers where contributions, while not semantically distant from their immediate peers, are part of a concentrated wave of scientific interest.

• •

  Trailblazing Neighborhoods (High Spatial / High Temporal): These are nascent, intellectually distinctive areas of inquiry that are also rapidly gaining momentum. Papers in these neighborhoods are positioned at the start of new, fast-growing research directions.

It is crucial to emphasize that this typology characterizes the context in which a paper is published, not the intrinsic attributes of the paper itself. A paper is thus described by the type of neighborhood it inhabits (e.g., ”a paper in a Trailblazing neighborhood”). This distinction is fundamental to our analysis, as it shifts the focus from a paper’s identity to its strategic position within the scientific landscape.

Figure 5 provides a stylized illustration of these four archetypes in semantic space. High spatial novelty corresponds to a large isolation radius at the time of publication ($t=0$). High temporal novelty is visualized by a significant decrease in the isolation radius between a prior period ($t=-3$) and the publication date, indicating a rapid densification of the paper’s intellectual neighborhood.

A key feature of our framework is the distinction between static isolation and its rate of change. One might consider using spatial novelty at $t=0$ and spatial novelty at $t=-3$ as the two dimensions. However, for a paper with high spatial novelty, its novelty three years prior is also likely to be high, providing little additional information. Our temporal novelty score, by measuring the change in isolation, is designed to capture this dynamic dimension directly. It effectively differentiates between an isolated paper in a static field (Outlying neighborhoods) and an isolated paper in a field that is rapidly forming around it (Trailblazing neighborhoods).

An analysis of the joint distribution of the two novelty scores reveals important structural features of the scientific landscape.

Our first analysis examines the likelihood of a paper becoming highly cited, defined as reaching the top 25% of citations within its publication-year-subfield cohort. The results, presented in Figure 8, reveal a clear and powerful pattern: temporal novelty is the primary predictor of citation success.

Increasing temporal novelty from low to high dramatically increases a paper’s probability of being highly cited. Among papers with low spatial novelty, papers in Trendy neighborhoods are 1.6 times more likely to be highly cited than papers in Consolidating neighborhoods (29.1% vs. 18.1%). Similarly, among papers with high spatial novelty, papers in Trailblazing neighborhoods are 1.65 times more likely than papers in Outlying neighborhoods (32.5% vs. 19.7%). This high-temporal-novelty advantage of over 10 percentage points is substantial and consistent across both levels of spatial novelty.

In contrast, the effect of spatial novelty is modest. Moving from low to high spatial novelty increases the high-citation probability by only 1.6 percentage points for low-temporal-novelty papers and 3.4 percentage points for high-temporal-novelty papers. While statistically significant⁷⁷7Detail numbers appear in Table 23, Appendix E.1., this effect is an order of magnitude smaller than that of temporal novelty.

This pattern suggests that positioning a contribution within a dynamic research frontier confers a powerful attentional advantage. Papers that engage with rapidly evolving conversations benefit from the field’s momentum, the influx of new researchers, and the dense citation networks that form around emerging topics.

The relationship between novelty and impact reverses when we examine disruption. As shown in Figure 9, spatial novelty emerges as the dominant predictor of disruptive influence.

Increasing spatial novelty from low to high nearly doubles a paper’s probability of being disruptive. Papers in Outlying neighborhoods are 1.9 times more likely to be disruptive than papers in Consolidating neighborhoods (22.4% vs. 11.8%), and papers in Trailblazing neighborhoods are 1.9 times more likely than papers in Trendy neighborhoods (20.2% vs. 10.5%). This high-spatial-novelty advantage is large and consistent across both levels of temporal novelty.

Conversely, temporal novelty is weakly negatively associated with disruptive potential. High-temporal-novelty papers are approximately 1.3 percentage points less likely to be disruptive than their low-temporal-novelty counterparts. This finding is conceptually intuitive: disruption, by definition, involves rendering prior work obsolete. Such a stark break is more likely to occur when a paper introduces a truly distinct perspective (high spatial novelty) rather than when it participates in a crowded, cumulative conversation at an evolving frontier (high temporal novelty).

Our framework identifies Trailblazing as a unique archetype that successfully navigates the trade-off between spatial and temporal novelty. While these are the rarest neighborhoods, their impact profile is exceptional. Papers in Trailblazing neighborhoods are not only strong performers on both individual impact dimensions (32.5% high-citation rate, 20.2% disruption rate), but they also exhibit a disproportionate likelihood of achieving both forms of impact simultaneously.

As shown in Figure 10, Papers in Trailblazing neighborhoods have a 5.5% probability of being both highly cited and disruptive. This is nearly three times the rate of papers in Consolidating neighborhoods (1.9%) and significantly higher than that of papers in Outlying are (4.0%) or Trendy neighborhoods (3.1%). This high dual-impact probability suggests a powerful synergistic effect: being both intellectually distinct (spatial) and well-timed (temporal) does not merely add to a paper’s impact, but amplifies its transformative potential, creating a rare pathway to both immediate relevance and lasting influence.

The core patterns identified in our analysis prove robust across a wide range of alternative specifications. As detailed in the appendix, the dominance of temporal novelty for achieving high-citation status and of spatial novelty for generating disruptive influence holds when using stricter citation thresholds (top 10%, top 1%) and when examining disruption rates within different subsamples of highly-cited papers (Appendices E.1 and E.3). An analysis of median citation counts adds further nuance: while confirming the primary role of temporal novelty, it also shows that spatial novelty provides a significant positive boost to citation levels, particularly for papers already high in temporal novelty (Appendix E.2). The synergistic advantage in achieving dual impact is similarly robust for papers in Trailblazing neighborhoods (Appendix E.4).

Furthermore, the framework’s integrity is confirmed by its insensitivity to key methodological choices. The fundamental relationships are preserved when using a more restrictive 75th percentile threshold to define our novelty archetypes (Appendix E.5). Crucially, field-by-field analyses demonstrate that the observed patterns hold within subfields, confirming that our findings represent general pathways to scientific influence rather than mere artifacts of disciplinary differences (Appendix E.6).

This body of evidence strongly supports our multidimensional framework. It challenges unidimensional approaches to novelty assessment by demonstrating that the pathways to scientific impact are not singular but plural, each requiring distinct intellectual strategies and offering different rewards.

Our analysis yields three primary discoveries that reshape the understanding of the novelty-impact relationship.

First, novelty is a multidimensional construct with a clear structural trade-off. We demonstrate that spatial novelty (intellectual distance from prior work) and temporal novelty (engagement with a dynamic research frontier) are empirically distinct and negatively correlated. This reveals a fundamental tension in research strategy: it is empirically difficult to be simultaneously highly original and perfectly timed within a burgeoning field.

Second, different dimensions of novelty predict different forms of impact. Temporal novelty is the primary predictor of citation success, as it positions a paper within an active conversation that attracts community attention. Spatial novelty, in contrast, is the primary predictor of disruptive influence, as it provides the intellectual distance necessary to render prior approaches obsolete. These are not interchangeable measures of “being novel” but distinct positioning patterns associated with different impact profiles.

Third, we identify a synergistic advantage associated with papers in Trailblazing neighborhoods. Papers that successfully navigate the trade-off to achieve both high spatial and high temporal novelty are disproportionately likely to achieve both high citations and high disruption. This suggests that distinctiveness and timeliness are not merely additive but interact to amplify a paper’s transformative potential.

Underlying these empirical findings is a central theoretical contribution: the decoupling of a paper’s identity from its position. Traditionally, novelty is treated as an intrinsic attribute of a contribution. Our framework reframes it as a positional good, defined by a paper’s relationship to its surrounding semantic neighborhood. This conceptual shift from an internal property to an external context is crucial. It moves the unit of analysis from the paper itself to the research locale it inhabits, allowing us to study how different types of intellectual positioning relate to different forms of scientific influence — how researchers select into and navigate different types of intellectual environments. This perspective provides a new mechanism for understanding scientific progress, one that accounts for both individual contributions and the ecological dynamics of the knowledge landscape in which they are embedded.

Our framework offers new tools for science policy and evaluation. By distinguishing between citation-generating and disruption-generating forms of novelty, our metrics reveal that current citation-heavy evaluation systems may systematically underweight the contributions of Outlying work. Papers in Outlying neighborhoods exhibit high disruptive potential but lower citation rates, suggesting their transformative value may be less visible through conventional metrics. Our framework provides a complementary lens for identifying different novelty profiles, enabling more nuanced assessment of what constitutes a “transformative” contribution across different impact dimensions.

Ultimately, our framework calls for a fundamental shift in how scientific contributions are discussed and evaluated. The common question, “Which papers are most novel?” is ill-posed. Our findings suggest it should be replaced with: “What type of novelty does this paper exhibit?” This reframing acknowledges that excellence in science takes multiple forms - each valuable, each measurable, and each essential for the collective progress of a thriving scientific enterprise.

Our study is subject to several limitations that point toward important directions for future research.