Predicting Drug Sensitivity in Cancer Cell Lines and Tumors with RNA Expression and Machine Learning

Research Impact

In a recent review article, Bender et al. 2026 showed that having a viable biomarker strategy for drugs can cut developmental costs by almost half (see figure below). This result highlights the importance of biomarker discovery in oncology drug development. Here, I present a simulation study aimed to promote the translation of in vitro drug sensitivity screens into tumor biomarker discovery, and to provide insights into the design of cancer cell line panel screen.

Fig.1b from Bender et al. 2026.

Results

In my previous post, I aligned the gene expression data between CCLE cancer cell lines and TCGA tumors using Celligner and a domain adversarial autoencoder (AE). In this new post, I perform a simulation to answer a few questions important for oncology biomarker discovery. I select a few drugs from the DepMap PRISM drug screen and examine whether the aligned transcriptomes can be used to predict drug sensitivity for unseen cell lines and infer tumor drug responses. Here are the key questions and summaries:

1. How does the prediction accuracy compare between Celligner- vs. AE-aligned data? I select 6 drugs with different mechanism of action and potential predictability, and randomly split the cell lines into training and test sets for 10 times (10-fold cross-validation) to evaluate the prediction accuracy of drug sensitivity (AUC) using elastic net (EN) regression with 1000X bootstrapping. Hyperparameter tuning is performed to ensure that each model is optimized for each condition. The data show that Celligner-aligned data largely outperforms AE-aligned data in predicting drug sensitivity for held-out (unseen) cell lines. However, it is important to note that the AE model was trained on a smaller number of genes (4,000 high variance genes vs. ~9,000 filtered genes in Celligner), and that the AE model compressed gene expression into a 256-dimensional latent space, which may have affected accuracy. Fine-tuning the AE model and evaluating a larger number of drugs may result in different conclusion.

Figure 1. Comparing prediction accuracy between Celligner and AE aligned RNA expression. Prediction accuracy is quantified by Pearson r between predicted vs actual AUC (y-axis) across 6 drugs (x-axis). Data is represented as mean, and error bars show 95% confidence intervals (CI) across the 10 randomly sampled train-test split (CV repeats). Celligner generally performs better than AE but note the caveats outlined in the text above.

2. For an in vitro drug sensitivity screen, how many cell lines are required to achieve a good prediction accuracy? For each drug, 50 to 250 cell lines are randomly sampled 10 times to predict held-out cell lines using EN modeling. The results show that the prediction accuracy increases most strongly from 50 to 150 cell lines, and then gradually plateaus after 150 cell lines. This suggests that a panel of 150 cell lines may be sufficient to represent the pool for drug sensitivity prediction - but again, this may depend on the drug and its mechanism of action.

Figure 2. Prediction accuracy vs cell line panel size. Held-out test samples’ prediction accuracy is quantified by Spearman r (y-axis) across different panel sizes (x-axis) sampled randomly. Data is represented as mean, and error bars show the standard deviation (SD) across CV repeats. For most drugs, accuracy grows more sharply from 50-150 samples, and starts to plateau passing that.

3. Is it better to concentrate the panel screen in one cancer lineage, or spread across different lineages? The same random sampling approach is applied to select cell lines from either a single lineage or across multiple lineages. The results show that for most drugs, a lineage-spread panel screen performs better than a lineage-concentrated panel screen, suggesting that a diverse panel of cell lines may be more informative. However, some drugs with lineage-specific mechanisms of action may benefit from a lineage-concentrated panel screen.

Figure 3. Prediction accuracy vs lineage composition. Held-out test samples’ prediction accuracy is quantified by Spearman r (y-axis) across drugs and aligning methods (x-axis), with either panels sampled from randomly-selected single lineage or across lineages (n = 30 cell lines). Mean and 95% CI are shown. This data suggests that cross-lineage drug sensitivity profiling may be more effective than single-lineage profiling, but this may apply only when the drug target is not strongly lineage-dependent.

4. Can we predict tumor drug responses using cell line panel screen, and discover tumor sensitivity biomarkers? The EN model trained on cell line panel screen data is applied to predict drug responses for >11k TCGA tumors across multiple indications. The EN model correctly identifies RB1, CCNE1 and CDKN2A as top biomarkers for CDK4/6 inhibitor palbociclib from the cell panel. When applied to tumor data, the predicted AUC of selected drugs significantly correlates with the expression of their known biomarkers, including IRF6 for erlotinib, CCNE1 for palbociclib, and DUSP6 for trametinib. As uncertainty quantification is important for ML prediction, the AUC intervals are also quantified by both bootstrapping (EN coefficients) and held-out conformal intervals.

Figure 4. EN top predictors of palbociclib AUC in DepMap cell lines. Top sensitive and resistance predictor genes of palbociclib drug response are plotted with their corresponding median EN coefficients. RB1, CCNE1 and CDKN2A are well-known biomarkers for palbociclib, suggesting that the model correctly identifies the biomarkers.

Figure 5. TCGA tumors: CCNE1 expression (x-axis) vs predicted palbociclib AUC (y-axis) by AE (left) or Celligner (right).

Figure 6. TCGA tumors: DUSP6 expression (x-axis) vs predicted trametinib AUC (y-axis) by AE (left) or Celligner (right).

Figure 7. TCGA tumors: IRF6 expression (x-axis) vs predicted erlotinib AUC (y-axis) by AE (left) or Celligner (right).

In summary, this simulation experiment suggests that with the proper domain alignment algorithm, cell line sampling and ML modeling, cell line panel screen data can be translated into biomarker insights for tumor drug responses.

I am a US-based independent consultant specialized in NGS, omics, statistical & ML analysis. Contact me via LinkedIn if you think you have a project that I can help with! https://www.linkedin.com/in/jay-cy-chung/

Additional Notes

Information on the 6 selected drugs:

palbociclib  CDK4/6 inhibitor       cell-cycle biomarker
trametinib   MEK inhibitor          MAPK-pathway biomarker
erlotinib    EGFR inhibitor         EGFR biomarker
olaparib     PARP inhibitor         BRCA1/2 / HR-deficiency biomarker
paclitaxel   tubulin / taxane       broad chemo, no single-gene biomarker (contrast arm)
bortezomib   proteasome inhibitor   mechanistically distinct 6th arm

EN hyperparameter tuning: since the alpha parameter in EN controls the mixing ratio between Ridge (L2) and Lasso (L1) regression, and that the Celligner or AE aligned expression have significantly different dimensions, alpha is assumed be be a structural hyperparameter that is domain-specific. Hence, the alpha parameter is tuned separately for Celligner vs AE data by cross-validation (CV). Once alpha is selected, the lambda parameter is tuned within each drug, sample size and repeat, so that penalization is optimized for each condition (e.g. smaller sample size might prefer larger penalty).

Disclosures

The code was written with the aid of Claude Opus 5.5. The accuracy of the code was verified by the author.

References

Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., Marchand, M., & Lempitsky, V. (2016). Domain-adversarial training of neural networks. Journal of Machine Learning Research, 17(59), 1–35.

Ghandi, M., Huang, F. W., Jané-Valbuena, J., Kryukov, G. V., Lo, C. C., McDonald, E. R., et al. (2019). Next-generation characterization of the Cancer Cell Line Encyclopedia. Nature, 569(7757), 503–508. https://doi.org/10.1038/s41586-019-1186-3

He, D., Liu, Q., Wu, Y., & Xie, L. (2022). A context-aware deconfounding autoencoder for robust prediction of personalized clinical drug response from cell-line compound screening. Nature Machine Intelligence, 4(10), 879–892. https://doi.org/10.1038/s42256-022-00541-0

Hoadley, K. A., Yau, C., Hinoue, T., Wolf, D. M., Lazar, A. J., Drill, E., et al. (2018). Cell-of-origin patterns dominate the molecular classification of 10,000 tumors from 33 types of cancer. Cell, 173(2), 291–304. https://doi.org/10.1016/j.cell.2018.03.022

Warren, A., Chen, Y., Jones, A., Shibue, T., Hahn, W. C., Boehm, J. S., Vazquez, F., Tsherniak, A., & McFarland, J. M. (2021). Global computational alignment of tumor and cell line transcriptional profiles. Nature Communications, 12, 22. https://doi.org/10.1038/s41467-020-20294-x