Explaining the Imperfect: How do LLMs Respond to Smelly Code?

Code smells are indicators of suboptimal design or implementation that contribute to technical debt, impairing software comprehensibility and maintainability. While Large Language Models (LLMs), particularly Chat LLMs like GPT-4o, are increasingly adopted by the developer community, their ability to handle such code imperfections remains under-explored. Existing studies show that LLMs can accurately define code smells when explicitly queried. However, it is unclear whether this knowledge translates into their code explanation capabilities. In this study, we perform a detailed empirical analysis of both general-purpose Chat LLMs (GPT-4o, GLM-4) and specialized code LLMs (CodeT5+, CodeQwen1.5) to investigate their sensitivity to code smells. We tasked these models with generating method-level summaries for Java code containing smells across three categories: Structural Complexity, Data/Type Smells, and Expression Clarity. Our results reveal that Chat LLMs exhibit limited sensitivity to code smells, often prioritizing fluent and polite explanations over identifying underlying quality issues. Both generalist and specialized models tend to generate consistent summaries regardless of the presence of smells, effectively masking potential code risks. We conclude that future LLMs require a more nuanced awareness of diverse code characteristics to effectively assist developers in code comprehension and maintenance.