Interpretable Link Prediction in AI-Driven Cancer Research: Uncovering Co-Authorship Patterns

Artificial intelligence (AI) is transforming cancer diagnosis and treatment. The intricate nature of this disease necessitates the collaboration of diverse stakeholders with varied expertise to ensure the effectiveness of cancer research. Despite its importance, forming effective interdisciplinary research teams remains challenging. Understanding and predicting collaboration patterns can help researchers, organizations, and policymakers optimize resources and foster impactful research. We examined co-authorship networks as a proxy for collaboration within AI-driven cancer research. Using 7,738 publications (2000-2017) from Scopus, we constructed 36 overlapping co-authorship networks representing new, persistent, and discontinued collaborations. We engineered both attribute-based and structure-based features and built four machine learning classifiers. Model interpretability was performed using Shapley Additive Explanations (SHAP). Random forest achieved the highest recall for all three types of examined collaborations. The discipline similarity score emerged as a crucial factor, positively affecting new and persistent patterns while negatively impacting discontinued collaborations. Additionally, high productivity and seniority were positively associated with discontinued links. Our findings can guide the formation of effective research teams, enhance interdisciplinary cooperation, and inform strategic policy decisions.

💡 Research Summary

This paper investigates how collaborative patterns among researchers in artificial‑intelligence‑driven cancer studies can be predicted and interpreted. Using Scopus, the authors extracted 7,738 publications spanning 2000–2017 that contain AI‑related keywords (e.g., “machine learning”, “deep learning”) together with “cancer”. From these records they built 36 overlapping co‑authorship networks, each representing a three‑year sliding window (e.g., 2000‑2002, 2001‑2003, …). Within each window, every pair of authors is labeled as a “new link” (first co‑authorship in that window), a “persistent link” (co‑authorship existed in the previous window and continues), or a “discontinued link” (co‑authorship existed previously but disappears).

Feature engineering combines two major streams. Attribute‑based features capture individual author characteristics: publication productivity (number of papers in the window), career length (years since first publication), institutional size (total output of the author’s affiliation), and primary discipline (medical, computer science, biology, etc.). The authors also compute a “discipline similarity” score by representing each author’s disciplinary distribution as a vector and measuring cosine similarity; this metric later proves to be the most influential predictor. Structure‑based features describe the relational context of a dyad: common neighbors, Jaccard similarity, Pearson correlation of adjacency vectors, clustering coefficient (triadic closure), differences in betweenness, closeness, and PageRank centralities.

Four supervised classifiers are trained to predict the three link states: logistic regression, support vector machine, XGBoost, and random forest. Because the data are heavily imbalanced (far more non‑links than links), the authors apply a hybrid resampling strategy that combines SMOTE oversampling of the minority class with random undersampling of the majority class. Model performance is evaluated primarily by recall (to capture as many true collaborations as possible) and secondarily by precision and F1‑score. Random forest consistently achieves the highest recall across all three link types (new = 0.78, persistent = 0.81, discontinued = 0.73), outperforming the other algorithms.

Interpretability is addressed with Shapley Additive Explanations (SHAP). Global SHAP plots reveal that discipline similarity contributes positively to the formation and maintenance of collaborations (high SHAP values for new and persistent links) but contributes negatively to link termination (low SHAP values for discontinued links). In contrast, high productivity and long career length have positive SHAP values for discontinued links, indicating that prolific, senior researchers are more likely to drop existing partners—perhaps because they shift focus to new projects or attract new collaborators. Structural features such as common neighbors and clustering coefficient generally support link persistence, yet their impact diminishes when discipline similarity is low, highlighting an interaction effect between topical alignment and network cohesion. Betweenness centrality differences exert a negative influence on link formation, suggesting that authors occupying very different positions in the network find it harder to start collaborations.

From a policy perspective, the findings suggest concrete actions. Institutions should design matchmaking programs that prioritize pairs with high disciplinary overlap, and they should organize interdisciplinary workshops to raise similarity scores across fields. Funding agencies could incorporate discipline similarity and network diversity into grant evaluation criteria to encourage truly cross‑disciplinary teams. For senior, highly productive scholars, targeted incentives—such as seed funding for collaborations with junior or out‑of‑discipline partners—could mitigate the tendency to abandon existing ties and promote knowledge diffusion.

The authors acknowledge several limitations. The dataset is limited to Scopus and English‑language venues, potentially omitting relevant regional publications. The three‑year sliding window may smooth over short‑term fluctuations in collaboration dynamics. Moreover, the study relies solely on quantitative network and bibliometric features; incorporating textual topic modeling (e.g., LDA) or sentiment analysis of acknowledgments could enrich the understanding of motivational factors behind collaboration.

In conclusion, this work provides a robust, interpretable machine‑learning framework for predicting co‑authorship evolution in AI‑driven cancer research. It demonstrates that disciplinary similarity is a pivotal driver of both new and sustained collaborations, while productivity and seniority paradoxically correlate with partnership dissolution. By making the predictive models transparent through SHAP, the study equips researchers, administrators, and policymakers with actionable insights to foster more effective interdisciplinary teams, ultimately accelerating progress against cancer.