Detection of metadata manipulations:
Finding sneaked references $\boldsymbol{\mathghost}$ in the scholarly literature

Visual inspection of several PDF files of the International Journal of Innovative Science and Research Technology (IJISRT), and their corresponding Crossref json records reveals that, in some cases, references are sneaked in at the end of the Crossref reference-list attribute.

On 24 July 2024, Cristian Consonni alerted some of the authors about an entry of the Problematic Paper Screener (Cabanac \BOthers., \APACyear2022) that highlighted tortured phrases in a certain IJISRT article (DOI: 10.38124/ijisrt/ijisrt24apr2410 – PubPeer). He further noted that the article had 237 citations, which was unusually high for an article published in April 2024, only 3 months earlier. Further inspection revealed that there were a significant number of other articles in IJISRT with a seemingly disproportionate number of citations and that most—if not all—citations had come from the same journal. As a result of this discovery, we sought citations in the actual text of the citing articles in vain, which indicates a pattern of sneaked references.

On the same day (24 July 2024), we queried Dimensions for all articles in IJISRT and retrieved the resulting CSV file, including the list of corresponding DOIs. For each retrieved DOI, the corresponding PDF file was downloaded from the publisher website (29 August 2024). Additionally, for all DOIs, Crossref records were downloaded from Crossref using the relevant API (28–29 August 2024). This served as a development dataset, and on 25 November 2024 we downloaded the final dataset presented here.

In this final dataset, the observed sneaked references are always benefiting papers with DOIs prefixed with 10.38124/ijisrt. This prefix identifies the International Journal of Innovative Science and Research Technology (IJISRT). All observed sneaked references appear in Crossref records after the expected references, as an irrelevant addendum to the reference list. We used these two properties (journal-level self-citation and position at the bottom of the sneaked references in Crossref metadata, reference-list attribute) to study to which extent sneaked references occur.

The idea is to extract a reference list from the PDF files for them to be compared with the ones registered with Crossref. Extracting the reference list from a PDF file can be done using a tool that transforms PDF files into XML files. In XML format, the reference list is clearly identified and can be automatically analysed.

Our process was the following for each collected DOI:

• •

  The reference list $\mathcal{R}_{\_}{C}$ registered with Crossref is built from the json file provided by Crossref. In the following, $Last_{\_}{C}$ denotes the last element of the list $\mathcal{R}_{\_}{C}$.

• •

  A reference list, $\mathcal{R}_{\_}{G}$ (in XML format) is extracted from the PDF file using Grobid (\APACcitebtitleGROBID, \APACyear2008–2023) (default configuration). Unfortunately, Grobid, while being quite reliable, sometimes skips some references. This results in missing references in $\mathcal{R}_{\_}{G}$. Items might be missing in bulk, either at the beginning, end or middle of the reference list. In very exceptional cases, Grobid inserts hallucinated references: $\mathcal{R}_{\_}{G}$ might contain references that do not appear in the PDF. We spotted cases where Grobid added the biography of an author as the last item of $\mathcal{R}_{\_}{G}$. This can happen when the bibliography appears, in the PDF, just after the last entry of the reference section (see the left panel in Figure 3). Nevertheless, we’ll use the last reference of $\mathcal{R}_{\_}{G}$, which we denote $Last_{\_}{G}$.

Comparing $Last_{\_}{G}$ to $Last_{\_}{C}$ gives information about both sneaked references and Grobid’s capability to identify correctly the last reference in the PDF’s reference list. Let us consider the three following cases:

• Case 1.

  If $Last_{\_}{C}=Last_{\_}{G}$ we conclude that $\mathcal{R}_{\_}{C}$ is correct with regards to the PDF version. The Crossref record does not contain any sneaked references.

• Case 2.

  If $\exists r\in\mathcal{R}_{\_}{C}$ such that $r=Last_{\_}{G}\land r\neq Last_{\_}{C}$ we conclude that references appearing after $r$ in $\mathcal{R}_{\_}{C}$ are sneaked references, forming a list denoted $\mathcal{L}_{\_}{\boldsymbol{\mathghost}}$. In the specific dataset of 10.38124/ijisrt, close analysis of $\mathcal{L}_{\_}{\boldsymbol{\mathghost}}$ instances reveal that, from time to time, the first items are not sneaked references. This happens when Grobid omits to extract from the PDF the end of the reference section, resulting in a truncated $\mathcal{R}_{\_}{G}$. Considering that in all inspected cases, all the sneaked references concerned DOIs starting with $10.38124$, we decided to remove from $\mathcal{L}_{\_}{\boldsymbol{\mathghost}}$ all elements preceding the first appearance of a DOI starting with $10.38124$. In other words, we skipped the references at the top of the list when they are not prefixed by $10.38124$. Without this cleaning operation, some of the legitimate references would have been wrongly considered as sneaked references.

• Case 3.

  If $\nexists r\in\mathcal{R}_{\_}{C}$ such that $r=Last_{\_}{G}$ we can conclude that $Last_{\_}{G}$ is an artifact created by Grobid. $Last_{\_}{G}$ is a hallucinated reference that does not appear, neither in $\mathcal{R}_{\_}{C}$ nor in the PDF file. In that case no conclusion can be drawn. Nevertheless, in the specific dataset of 10.38124/ijisrt, a close analysis reveals that—sometimes—sneaked references can be found at the end of $\mathcal{R}_{\_}{C}$. Again, in all inspected cases, the sneaked references are benefiting DOIs starting with $10.38124$. We decided to build $\mathcal{L}_{\_}{\boldsymbol{\mathghost}}$ with all elements of $\mathcal{R}_{\_}{C}$ appearing after the last element that has not a DOI starting with $10.38124$. Without this backward check (iterating over the list from the bottom to the first non $10.38124$ DOI), some of the sneaked references would have been wrongly considered as sneaked references.

Figure 3 illustrates a Case 3 situation. For a DOI like this, the list $\mathcal{L}_{\_}{\boldsymbol{\mathghost}}$ is built from the trailing references with DOIs starting with $10.38124$. This method could be used for every DOI. Identifying Case 1 and Case 2 is a way to evaluate how precise the extraction of sneaked references using Grobid can be.

The rationale behind this method is that the field unstructured of a particular reference entry in a Crossref record reflects closely the text of this reference in the PDF file. As a consequence, the character string $s$ found at this field must appear in the text $\mathcal{T}$ extracted from the PDF file. There is even no need to restrict the search of $s$ to the reference section. As shown in Section 2.2, identifying correctly the reference section is challenging. Unstructured references are quite long, thus the search of $s$ in $\mathcal{T}$ is unlikely to generate false positive.

More specifically, the following steps were performed for every DOI:

1. 1.

   The reference list $\mathcal{R}_{\_}{C}$ registered with Crossref is built from the json file provided by Crossref. In the following, $s$ denotes an element of the list $\mathcal{R}_{\_}{C}$.

2. 2.

   The full text $\mathcal{T}$ is extracted from the PDF file using the pypdf Python library.

3. 3.

   $\forall s\in{\mathcal{R}_{\_}{C}}$, a search of $s$ in $\mathcal{T}$ is performed. The goal is to identify $s^{\prime}$ the substring of $\mathcal{T}$ that is the closest to $s$ according to $\delta(s,s^{\prime})$ the normalized Levenstein distance¹⁴¹⁴14partialratio method from the RapidFuzz Python library. The similarity $\delta(s,s^{\prime})$ is ranging from 0 (totally different) and 100 (entirely similar).

4. 4.

   If $\delta(s,s^{\prime})<60$ this means that no character string $s^{\prime}$ highly similar to $s$ could be found in $\mathcal{T}$. This most probably happens when the reference $s\in\mathcal{R}_{\_}{C}$ does not exist in the document, revealing that $s$ is a sneaked reference. On the contrary, if $\delta(s,s^{\prime})\geq 60$ a character string $s^{\prime}$ quite similar to $s$ exists in $\mathcal{T}$. In that case $s$ is not a sneaked reference. The $60$ threshold was set experimentally, after manual examination of several cases.

We consider here the $3,132$ records with $\mathcal{R}_{\_}{C}\neq\emptyset$ and $\mathcal{R}_{\_}{G}\neq\emptyset$ that contain a total of $78,736$ sneaked references. These records are distributed among the three identified cases (see 2) as follows:

• Case 1.

  $331$ DOIs ($10.5\%=\nicefrac{{331}}{{3,132}}$) with no sneaked references where correctly identified because $Last_{\_}{C}=Last_{\_}{G}$.

• Case 2.

  When $\exists r\in\mathcal{R}_{\_}{C}$ such that $r=Last_{\_}{G}\land r\neq Last_{\_}{C}$ then $\mathcal{L}_{\_}{\boldsymbol{\mathghost}}$ is composed of references appearing after $r$ in $\mathcal{R}_{\_}{C}$. A cleaning operation (item) might be needed.

  • –

    No Cleaning needed: For $1,788$ ($\%=\nicefrac{{1,788}}{{3,132}}$) DOIs, $\mathcal{L}_{\_}{\boldsymbol{\mathghost}}$ was correct. This represents a total of $46,297$ ($58.8\%=\nicefrac{{46,297}}{{78,736}}$) sneaked references.

  • –

    Cleaning needed: For $840$ DOIs ($\%=\nicefrac{{840}}{{3,132}}$), $\mathcal{L}_{\_}{\boldsymbol{\mathghost}}$ contains potential false positives: references that would have been classified as sneaked without the cleaning operation. This represents a total of $2032$ references ($2.2\%=\nicefrac{{2032}}{{78,736}}$).

• Case 3.

  For $173$ DOIs, $\nexists r\in\mathcal{R}_{\_}{C}$ such that $r=Last_{\_}{G}$. A backward check (item) might be needed to identify sneaked references. For the $173$ instances of this case, the backward check identifies sneaked references. Without this check $3,176$ sneaked references would have been undetected ($4\%=\nicefrac{{3176}}{{78,736}}$).

Method $\mathcal{M}_{\_}1$ relies to a high extent on the tool use to extract the reference list from the PDF. We did implement this method using Grobid. Performances of the method is thus dependant of the ability of Grobid to accurately extract the reference list. Since this task is not trivial, and Grobid occasionally makes mistakes, we had to use additional assumptions about the sneaked references to get reliable results: sneaked references appear after the last genuine reference extracted from the PDF file. This means that this method might not generalize easily to other instances of the sneaked references problem.

Method $\mathcal{M}_{\_}2$ is not dependant on identifying the reference list in the PDF file and does not make any assumptions about where sneaked references are. As such, this method should generalize better than the first one. Nevertheless, in some corner cases in the text extracted from the PDF, additional text fragments like headers or footers might appear in the middle of a reference, thus making the reference identification impossible.

One common drawback of both methods is that the field unstructured must be correctly deposited with Crossref by publisher for the methods to work properly: one could imagine cases where only reference DOIs are provided.

For every DOI with references in the metadata and available PDF, we compared the numbers of sneaked references reported by both methods (see Table 1). For this dataset, among $3,186(=2,953+233)$ compared DOIs, the methods disagreed in 233 (7.3%) cases. Among these, for only 11 DOIs a difference greater than 10 is observed for the number of sneaked references reported. This explains why the total numbers of sneaked references detected by the methods differ only by $0.9\%$ ($\nicefrac{{80,909-80,205}}{{80,205}}$).

It seems that most discrepancies are due to cases where the first method underestimated the number of sneaked references.

Comparing the list lengths, (adapting (Besançon \BOthers., \APACyear2024) that uses HTML reference lists and Crossref reference lists) would give $84,270$ sneaked references. This is an overestimation of $5,534$ ($7\%=\nicefrac{{(84,270-78,736)}}{{78,736}}$) of the total number of sneaked references.

For some DOIs the error can be quite high (max = 465). Relying solely on lists length comparison will generates many false positive.

This an important drawback of method $\mathcal{M}_{\_}0$ when implemented using Grobid. It suggests that reference lists extracted using Grobid tend to be shorter than the ones actually existing in documents.

Using a length comparison method will thus overestimate the number of sneaked references. Using either the last element method $\mathcal{M}_{\_}1$ or the raw comparison method $\mathcal{M}_{\_}2$ is a more accurate way detect sneaked references.

The next section provides a detailed description of the results obtained using the systematic comparison of Grobid output vs Crossref records ($\mathcal{M}_{\_}1$).

• •

  The corpus is composed of $4,077$ DOIs prefixed by 10.38124/ijisrt.

• •

  For $3,222$ DOIs, a non-empty reference list $\mathcal{R}_{\_}{C}$ was downloaded from api.crossref. An empty result can reflect the fact that a document does not contain any reference list. But it also happens when the publisher did not register any reference list for this particular DOI. This means that the references listed in the real document are lost \faSkullCrossbones (see Besançon \BOthers., \APACyear2024). Despite being present in the PDF, they are not registered with Crossref and are not credited to the cited document. Consequently, for $4,077-3,222=855$ DOIs the number of sneaked references is zero, as strictly no references are registered with Crossref.

• •

  For $3,940$ DOIs, a non-empty reference list $\mathcal{R}_{\_}{G}$ has been extracted by Grobid from the PDF files. An empty list is generated either when the document does not contain any reference list or when Grobid failed to identify the reference section.

• •

  The records for $3,132$ DOIs have both $\mathcal{R}_{\_}{C}\neq\emptyset$ and $\mathcal{R}_{\_}{G}\neq\emptyset$. They contain a total of $78,736$ sneaked references.

• •

  Overall, the $80,205$ sneaked references were found in $2,782$ Crossref records. The number of sneaked references in a single paper ranges from $1$ to $71$, with an average of $28.83$ sneaked references par paper (see distributions in Figure 4).

• •

  The sneaked references are benefiting $2,703$ different DOIs. The most extreme case is a single DOI benefiting from $6,059$ undue citations.

• •

  DOIs with sneaked references were created with Crossref between March 2024 and November 2024.

A total of $80,205$ sneaked references are benefiting $2,782$ different DOIs. The average count of undue citations per benefiting DOI is $28.83$. Nevertheless, the distribution is very unbalanced as show in Figure 5. The overwhelming majority of DOIs ($n=2,469$) are credited with only a single undue citation. On the other hand, a small number of DOIs benefit from a significant number of undue citations.

The $30$ DOIs that benefit the most from sneaked references are shown in Figure 6. The figure also shows the count of undue citations that these DOIs benefit from. The DOI benefiting the most (10.38124/ijisrt/ijisrt24apr651) from sneaked references is credited with $6,059$ undue citations. Consequently, this particular DOI is incorrectly credited with $1.8k$ and 1.7k citations by OpenAlex and Dimensions, respectively (See Figure 1 and Figure 2). This shows that some of the sneaked references effectively made their way through the counting processes onto some scientometric platforms.

The information available from Crossref includes the date on which a DOI was first registered: creation date. For sneaked references, we decided to compare the creation date of the citing DOI and the creation date of the cited DOI.

As a result it can be said that sneaked references were first added in small numbers in April 2024. At that time only a handful of references were unduly added at registration time. The number of sneaked references increased little by little reaching a maximum of $71$ sneaked references per paper in November 2024.

The sneaked references are all benefiting the journal they are sneaked in and thus benefiting the publisher that registered them.

The time difference between the cited and the citing paper, for sneaked references, is often surprisingly small. It is also worth noting that sneaked references are not benefiting to cited papers in a very unbalanced way. Most of the papers are only benefiting from one single sneaked reference, while a few are benefiting for hundreds of undue citation.

In the light of these data, it is hard to find definitive evidence that differentiates intentional manipulation from genuine malfunctions in the meta-data registration process.

Nevertheless, awkward metadata registered with Crossref might help to identify venues where references are sneaked in. The next section investigates this hypothesis.

We hypotheses that sneaked references appear together with duplicated references. Therefore, we try to detect groups of duplicated references in the hope to circumscribe sneaked references.

A bibliography should not contain the same DOI multiple times. The approach we adopted is to consider only the DOI of references to identify duplicates. Thus, two references in a same citing article pointing to the same DOI is considered as a duplicate reference. Working at the DOI level to spot duplicates is imperfect. We noticed that some DOIs occur multiple times in a Crossref record of a bibliography that does not feature duplicates. This can happen when a DOI of a book is used to identify different chapters of this book. For this reason, we excluded books and book chapters from our analysis.

We used the latest Crossref snapshot downloaded on 23 November 2023. It contains a total of 991,206,078 reference entries including duplicates. 3,755,847 ($0.38\%$) of these are duplicated 1+ times. Thus, the dump contains a total of 986,772,474 distinct references. Overall, 4,433,404 ($0.45\%$) reference entries are duplicates; and there is an average of 1.18 duplicates per duplicated reference.

Our goal is to identify entities that benefit the most from duplicated references. The entities we consider are either publishers, authors, articles, or journals. Let us introduce the following notations:

• •

  Let $D$ be the set of documents (limited to articles),

• •

  $A$ be the set of authors,

• •

  $P$ be the set of publishers

• •

  and $J$ be the set of journals.

Authors are authoring documents that are published in journals and documents are referencing other documents. To denote this, we’ll use the following notations:

• •

  For $a\in A$ and $d\in D$, “$a\text{\hskip 3.0pt\raisebox{-2.0pt}{{\char 29\relax}}\hskip 3.0pt}d$” means: $a$ is author of $d$

• •

  For $j\in J$ and $d\in D$, “$d\text{\hskip 3.0pt\faHandshake[regular]\hskip 3.0pt}j$” means: $d$ is published in $j$

• •

  For $(d_{\_}1,d_{\_}2)\in D^{2},n\in\mathbb{N}$,
  “$d_{\_}1\xrightarrow{\;n\;}d_{\_}2$” means: $d_{\_}1$’s metadata contains exactly $n$ references to $d_{\_}2$.
  “$d_{\_}1\xrightarrow{\;\;\;}d_{\_}2$” means: $d_{\_}1$’s metadata contains at least one reference to $d_{\_}2$.

We also consider the following sets:

• •

  $D_{\_}j$ the set of the documents published in journal $j\in J$: $D_{\_}j=\left\{d\in D\mid d\text{\hskip 3.0pt\faHandshake[regular]\hskip 3.0pt}j\right\}$.

• •

  $R_{\_}d$ the set of the references of document $d\in D$: $R_{\_}d=\left\{\left(d,r\right)\in D^{2}\mid d\xrightarrow{\;\;\;}r\right\}$.

• •

  $R^{+}_{\_}d$ the set of references of document $d\in D$, that are duplicated 1+ times:

  $R^{+}_{\_}d=\left\{\left(d,r\right)\in R_{\_}d\mid n\in\mathbb{N},n>1,d\xrightarrow{\;n\;}r\right\}$. Note that $R^{+}_{\_}d\subseteq R_{\_}d$.

• •

  $C_{\_}d$ the set of the documents citing document $d\in D$:$C_{\_}d=\left\{c\in D\mid c\xrightarrow{\;\;\;}d\right\}$.

To measure how much a registered DOIs contains duplicated references or how much a paper benefit from duplicated references, the following measure are defined:

• •

  $\operatorname{NbRef}\left(d_{\_}1,d_{\_}2\right)$ denotes the number of times $d_{\_}1\in D$ contains a reference to $d_{\_}2\in D$.

• •

  $Benef^{+}(d)=\sum_{\_}{c\in C_{\_}d}\left(\operatorname{NbRef}\left(c,d\right)-1\right)$ is the number of duplicated references benefiting $d$ and
  $Benef(d)=\left|\left\{c\in C_{\_}d\mid\operatorname{NbRef}\left(c,d\right)>1\right\}\right|$ is the number of 1+ duplicated references $d$.

• •

  $NbRefDup^{+}(d)=\sum_{\_}{\left(d,r\right)\in R^{+}_{\_}d}\left(\operatorname{NbRef}\left(d,r\right)-1\right)$ is the number of duplicated references in metadata registered for document $d$,
  $NbRefDup(d)=\left|R^{+}_{\_}d\right|$ is the number of reference duplicated 1+ times, in metadata registered for document $d$.

Some articles ‘benefit’ from an impressive number of duplications. The top 5 is shown in Table 2). Quite interestingly, the landing page (using doi.org) for the first row is currently a generic error page. Table 3 provide the top 5 papers for which metadata contains a lot of duplicated references. Again the landing page for the first DOI leads to a Page not Found error. Visual inspection of Crossref records for these five DOI reveal simple duplicated references without any obvious sneaked references.

To measure how much a journal does contain duplicated references, the following measures are computed:

• •

  $JourDup^{+}(j)=\sum_{\_}{d\in D_{\_}j}\sum_{\_}{\left(d,r\right)\in R^{+}_{\_}d}\left(\operatorname{NbRef}\left(d,r\right)-1\right)$ the number of duplicated references found in the metadata of a journal $j$.

• •

  $JourDup(j)=\left|\left\{d\in D_{\_}j|R^{+}_{\_}d\neq\emptyset\right\}\right|$ the number of documents registered for journal $j$ that contain at least one duplicated reference.

Table 4 lists journals for which the highest number of duplicated references are found. Such journals push a great amount of duplicated reference metadata to Crossref. It is important to note that, despite the name, SSRN Electronic Journal is a platform for pre-prints. The first position might be explained by the high number of papers the platform register with Crossref. In second position, the Journal of Behavioral Addictions did register an average of 37 duplicated references over $603$ articles. Again, close inspection of some examples reveals only simple duplicated references without any obvious sneaked references.

Previous work (Besançon \BOthers., \APACyear2024) showed that sneaked references are sometimes benefiting to particular authors. Thus, identifying authors that benefit from duplicated references may also be the ones that benefit from sneaked references.

We note $s_{\_}{1a}(j,a)$ the proportion of references found in journal $j$ that cite a paper authored by $a$:

$s_{\_}{1b}(j,a)$ refers to the proportion of duplicated references found in journal $j$ that cite a paper authored by $a$:

Hypothesising duplications occur randomly because they are mistakes, we should observe $s_{\_}{1a}(j,a)\sim s_{\_}{1b}(j,a)$. If $s_{\_}{1b}(j,a)$ is far greater than $s_{\_}{1a}(j,a)$ this mean that duplicated references in journal $j$ are benefiting in an abnormal proportion to author $a$.

That is why we compute $s_{\_}1(j,a)$ an estimation of the number of duplicated references in journal $j$ that are statistically speaking unexpected for authors $a$:

Computing this score (Table 5) can reveal statistical anomalies that may (or may not) reflect citation gaming.

The computed leaderboard features Harikrishna B. Jethva and Bhavesh Kataria together with the International Journal of Scientific Research in Science and Technology from the Technoscience Academy publisher. This case being the one described in (Besançon \BOthers., \APACyear2024). This result is coherent with our hypothesis that duplicated references might be correlated with sneaked references. At the time of the Crossref snapshot was created metadata were not yet corrected. Since then, when asked by Crossref, the publisher did correct the records and removed sneaked references.

Some other authors in this list are suspected to manipulate their $h$ index (e.g., P. S. Aithal¹⁵¹⁵15https://www.researchgate.net/post/Excessiveself-citationinhisresearchpaperswhichhasartificiallyinflatedhisH-indexscore).

We did not check all the articles published by the journal–author pairs of this leaderboard, and further analysis might give new interesting results. Nevertheless, the case of the pair International Journal of Laser Dentistry and A. L. McKenzie is of some interest. we indeed found sneaked references benefiting to A. L. McKenzie’s articles. For example, 10.5005/jp-journals-10022-1031 contains a duplicated sneaked reference to 10.1109/geoinformatics.2015.7378602). Deeper investigations reveal that these sneaked references are not resulting from intentional manipulations but most probably are the consequence of genuine errors. This sneaked reference might be unintentional as its seems that this journal always sent the same reference list for all the metadata of its articles, except for the $n$ first references of each list that are replaced by the $n$ references of the current published article. This reference list contains the expected list of references (found in the PDF file) but are always padded up to $155$ with the same set of sneaked references. This journal is no more active and for each and every published article $d$ the website (landing page) is providing a list of exactly $155$ references that are the ones registered at Crossref.

Hoping that a combination of methods might enable the detection and validation of sneaked references at scale, we compared the length of the reference list extracted from PDFs using Grobid ($\mathcal{M}_{\_}0$) with the references registered with Crossref. For articles published since 2000, the number of reference items identified by Grobid were compared to the number of references provided by Crossref. In order to account for a reported Grobid uncertainty rate of 0.05, we allowed for a margin of error, comparing the length of references identified by the full-text Grobid processing to 0.95 times the Crossref references count. A total of 4,172,499 articles out of 47,170,721 processed were found to have fewer references than 95 percent of the Crossref reference count.

In order to determine which publications had been added and which authors or journals had been inserted erroneously, we attempted to match the references extracted by Grobid with the identifiers supplied by Crossref. This approach turns out to be challenging, partially as a result of the inconsistency of reference formatting in the PDFs (many were missing DOIs) but also highlighted an error in the initial approach: references which are provided in supplementary attachments were not counted in the original Grobid-processed PDFs.

Although this attempt to systematically identify erroneous references was not effective, there remain a total of 1,564,408 publications with between 5 and 500 additional references reported to Crossref beyond those identified by Grobid processing. The possibility remains that this approach may work for a subset of the total dataset (see Section 2.6 for more details on the limitations of the precision of the extraction of references using Grobid).

We identified a new set of sneaked references that benefit a single journal: IJISRT. The sneaked references are registered with Crossref along with the metadata for this journal. As a result, sneaked references inflate citations counts for this journal and for some of its articles.

Crossref provides infrastructure for registering metadata for scholarly works, including a DOI. To use this infrastructure, organisations join Crossref as members, taking on related obligations¹⁶¹⁶16https://www.crossref.org/membership/terms/. At the most basic level, Crossref members are responsible for depositing accurate metadata for each content item they produce.

When a serious issue with the metadata is detected, Crossref contacts the member to investigate the situation and work with them to rectify the problem where applicable. In some rare cases, the member’s access to register new scholarly works or update their existing records might be temporarily suspended or their membership permanently revoked.

The Crossref records for $\sim{}2.7k$ DOIs from the International Journal of Innovative Science and Research Technology require corrections to remove the $\sim{}81k$ sneaked references. In November 2024, Crossref contacted the member responsible for the International Journal of Innovative Science and Research Technology to ask for an explanation. Based on the member’s replies to the enquiries, it was clear there was an intention to manipulate the citation record, and as such Crossref have started the process to revoke this organization’s membership.

More information on Crossref’s membership revocation process can be found here ¹⁷¹⁷17https://www.crossref.org/operations-and-sustainability/membership-operations/revocation/. The most up-to-date list of revoked Crossref members is available here.¹⁸¹⁸18https://docs.google.com/spreadsheets/d/1cCkdvtqEM1urmrUQZ4-LGzOmf5812aVkRFJc5UryHw/edit This member will appear on the list if their revocation is ratified by the Crossref board.

Crossref encourages the community to report cases via the dedicated ”metadata quality improvements” channel¹⁹¹⁹19https://community.crossref.org/c/tech-support/metadata-quality-improve/45 on its forum.

There are a number of possible sources of erroneous references, not all of which are nefarious. The identification of publications whose reference count fails to match the references actually listed in the references section (which may differ, in turn, from the in-text citations), is only the first, judgment neutral, step. There are some patterns detectable in the erroneous references, which hint at the source.

For example, in one situation there were several publications with a valid list of references which seemed to have been written on top of a constant, longer list of references in identical order. The number of total references was constant, while the number of valid references varied. This situation suggests to us an error in pasting, where a shorter list of references pasted over, rather than replaced, a longer list of references from an earlier metadata submission. In other cases, we found multiple identical references pasted at the end of the list of valid references. This behavior suggests a less technical source, and less benign intentions.

There are different methods for members to register scholarly metadata with Crossref. These range from plugins integrated into publishing platforms to registration forms with different metadata fields for members to fill. XML files can also by directly sent using HTTPS POST. On the member side, during the publishing process different actors might have different kind of access to the metadata records, providing different kinds of opportunities to sneak references in or register erroneous metadata.

Sneaked references remain one of many possibilities to practice citation gaming (see Section 1). More of such gaming will continue to prevail as long as academic value is tightly coupled to specific metrics.

Given the variety of reasons for erroneous references, there are multiple approaches that could be taken to improve the situation, which include improving the tools by which editors submit reference lists, the automated deduplication of reference lists after submission, and systematically cross-checking publications using Grobid in collections (proprietary or otherwise) which contain full-text PDFs. Deliberate efforts, which measure the rate of success of these approaches, are advisable.

Beyond this specific data set, the extent to which sneaked references are distorting citation counts is unknown. It might remain a very limited phenomenon but this needs to be verified by further investigations.

Identifying those journals or authors which have been most frequently associated with erroneous references, at scale, may allow us to identify the beneficiaries of sneaked references. This could act as a heuristic device to search for additional sneaked references (see Section 4.1), and to distinguish between erroneous and sneaked references.

Future work will attempt to identify erroneous references at a larger scale. We will continue to attempt to define patterns that can be used to flag sneaked references. This big data approach will help us determine whether the sneaked reference would have had a bibliometric effect, resulting in any increase in the Journal Impact Factor (or other journal-based metrics).