Exploring the Capabilities and Limitations of Small Language Models for Domain-Specific Tasks: A case study in Biomedical Question Answering

Recent advances in large transformer-based language models have yielded impressive performance on biomedical question answering, yet their computational cost and model size hinder deployment in resource constrained environments. This paper investigates whether a small, recurrent, domain-specific language model, built from bottom up and trained on curated biomedical corpora, can effectively model healthcare language and generalize to biomedical QA. We design a complete pipeline including a custom byte-pair encoding (BPE) tokenizer, a three-layer long short-term memory (LSTM) language model with tied embeddings, two-phase training (healthcare language modeling followed by QA-style fine-tuning), and a retrieval-augmented generation (RAG) framework based on BM25 retrieval over MedQuAD, BioASQ Task B, and PubMedQA. The proposed SLM achieves strong language modeling performance (Phase 1 test perplexity of 36.70, Phase 2 QA test perplexity of 19.37) on 4.6M trained tokens and generates fluent, medically styled text. However, despite the low perplexity, the model still struggles with semantic factuality and long range reasoning, even when provided with the relevant retrieved context. We observe frequent hallucinations, off-topic disease descriptions, and limited grounding in retrieved evidence. These findings suggest that small RNN-based language models, trained with next-token prediction on noisy QA corpora, are greatly suited for biomedical language modeling but are not sufficient enough for factually reliable biomedical QA, underscoring the importance of dataset alignment and supervised QA objectives.