top of page

AI Failure Modes in Regulatory Writing:
A Risk Awareness Framework

Sruthi Sampathkumar

Resources for the poster presented at the
AMWA Medical Writing & Communication Conference 2026

About the framework

The framework organizes 14 AI failure modes into
5 categories and maps each to its anticipated consequence in regulatory documents and to the writer competency needed to recognize and mitigate it. It was developed through a targeted review of AI, medical natural language processing, and clinical decision support literature (2023–2026), informed by regulatory writing practice.

Download the Poster (pdf)

Help validate the framework

Have you seen these failure modes in your own work?

Take a 2-minute survey
REFERENCES

  1. Topaz M, Roguin N, Gupta P, Zhang Z, Peltonen LM. Fabricated citations: an audit across 2.5 million biomedical papers. Lancet. 2026;407(10541):1779-1781. doi:10.1016/S0140-6736(26)00603-3

  2. Asgari E, Montaña-Brown N, Dubois M, et al. A framework to assess clinical safety and hallucination rates of LLMs for medical text summarisation. NPJ Digit Med. 2025;8(1):274. doi:10.1038/s41746-025-01670-7

  3. Jansen T, Liebenow LW, Mertens U, et al. Data extraction by generative artificial intelligence: assessing determinants of accuracy using human-extracted data from systematic review databases. Psychol Bull. 2025;151(10):1280-1306. doi:10.1037/bul0000501

  4. Bhattacharyya M, Miller VM, Bhattacharyya D, Miller LE. High rates of fabricated and inaccurate references in ChatGPT-generated medical content. Cureus. 2023;15(5):e39238. doi:10.7759/cureus.39238

  5. Walters WH, Wilder EI. Fabrication and errors in the bibliographic citations generated by ChatGPT. Sci Rep. 2023;13(1):14045. doi:10.1038/s41598-023-41032-5

  6. Topaz M, Zhang Z, Roguin N, Gupta P, Li Z, Peltonen LM. Biomedical reference generation remains unreliable across 26 large language models. arXiv. Preprint posted online September 14, 2026. doi:10.48550/arXiv.2609.14988

  7. Peters U, Chin-Yee B. Generalization bias in large language model summarization of scientific research. R Soc Open Sci. 2025;12(4):241776. doi:10.1098/rsos.241776

  8. Liu NF, Lin K, Hewitt J, et al. Lost in the middle: how language models use long contexts. Trans Assoc Comput Linguist. 2024;12:157-173. doi:10.1162/tacl_a_00638

  9. Modarressi A, Deilamsalehy H, Dernoncourt F, et al. NoLiMa: long-context evaluation beyond literal matching. In: Proceedings of the 42nd International Conference on Machine Learning. Vol 267. PMLR; 2025:44554-44570.

  10. Kobak D, González-Márquez R, Horvát EÁ, Lause J. Delving into LLM-assisted writing in biomedical publications through excess vocabulary. Sci Adv. 2025;11:eadt3813. doi:10.1126/sciadv.adt3813

  11. Ouyang S, Zhang JM, Harman M, Wang M. An empirical study of the non-determinism of ChatGPT in code generation. ACM Trans Softw Eng Methodol. 2025;34(2):42. doi:10.1145/3697010

  12. Atil B, Aykent S, Chittams A, et al. Non-determinism of "deterministic" LLM settings. arXiv. Preprint posted online August 6, 2024. Updated April 2, 2025. doi:10.48550/arXiv.2408.04667

  13. Nicholson C. Quantifying non deterministic drift in large language models. arXiv. Preprint posted online January 12, 2026. doi:10.48550/arXiv.2601.19934

  14. Sclar M, Choi Y, Tsvetkov Y, Suhr A. Quantifying language models' sensitivity to spurious features in prompt design or: how I learned to start worrying about prompt formatting. Paper presented at: 12th International Conference on Learning Representations; May 2024; Vienna, Austria.

  15. Hua A, Tang K, Gu C, Gu J, Wong E, Qin Y. Flaw or artifact? Rethinking prompt sensitivity in evaluating LLMs. In: Christodoulopoulos C, Chakraborty T, Rose C, Peng V, eds. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics; 2025:19889-19899. doi:10.18653/v1/2025.emnlp-main.1006

  16. Romanou A, Ibrahim M, Ross C, et al. Brittlebench: quantifying LLM robustness via prompt sensitivity. arXiv. Preprint posted online February 27, 2026. Updated April 6, 2026. doi:10.48550/arXiv.2603.13285

  17. Chen L, Zaharia M, Zou J. How is ChatGPT's behavior changing over time? Harv Data Sci Rev. 2024;6(2). doi:10.1162/99608f92.5317da47

  18. Jabbour S, Fouhey D, Shepard S, et al. Measuring the impact of AI in the diagnosis of hospitalized patients: a randomized clinical vignette survey study. JAMA. 2023;330(23):2275-2284. doi:10.1001/jama.2023.22295

  19. Rosbach E, Ganz J, Ammeling J, Riener A, Aubreville M. Automation bias in AI-assisted medical decision-making under time pressure in computational pathology. In: Palm C, et al, eds. Bildverarbeitung für die Medizin 2025. Informatik aktuell. Springer Vieweg; 2025. doi:10.1007/978-3-658-47422-5_27

  20. Chen S, Gao M, Sasse K, et al. When helpfulness backfires: LLMs and the risk of false medical information due to sycophantic behavior. NPJ Digit Med. 2025;8:605. doi:10.1038/s41746-025-02008-z

  21. Jain S, Park C, Viana M, Wilson A, Calacci D. Interaction context often increases sycophancy in LLMs. In: Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery; 2026:Article 793. doi:10.1145/3772318.3791915

  22. Clusmann J, Ferber D, Wiest IC, et al. Prompt injection attacks on vision language models in oncology. Nat Commun. 2025;16(1):1239. doi:10.1038/s41467-024-55631-x

bottom of page