LotBoNC: Novel Botnet Traffic Classification under Long-tailed Distributions

Botnets are a persistent cyber threat, leveraging global infrastructures to launch large-scale attacks. Yet, most existing classification methods are evaluated under balanced and closed-set assumptions, which fail to capture real-world conditions. In practice, botnet traffic is both long-tailed and open-world: unknown variants continually emerge, and rare threats are buried under dominant traffic, often evading detection. To reflect real-world conditions, we define a deployment-oriented setting where unlabeled traffic follows a long-tailed distribution, with dominant known classes in the head and rare novel botnet variants in the tail. We propose LotBoNC, a unified framework for encrypted traffic classification under long-tailed open-world conditions. LotBoNC first performs self-supervised pre-training to learn transferable representations, then applies entropy-regularized optimal transport to assign pseudo-labels aligned with estimated class priors. An EM-style loop iteratively refines prototypes and priors, improving class separation between frequent and rare categories. We evaluate LotBoNC on three public encrypted traffic datasets with diverse long-tailed scenarios. LotBoNC consistently outperforms prior state-of-the-art methods and accurately classifies known botnets and discovers unseen botnet variants in diverse, umbalanced open-world scenarios.