Development of a hybrid probabilistic key generation method using multi-prime Rivest-Shamir-Adleman and learning with errors to enhance Hill cipher security

This study focuses on the vulnerabilities of classical matrix-based encryption and asymmetric padding schemes within hybrid cryptosystems for securing high-throughput enterprise data streams. The Hill cipher algorithm is highly vulnerable to known-plaintext attacks and frequency distribution analysis, whereas the standard Rivest-Shamir-Adleman (RSA) padding scheme of public-key cryptography standards (PKCS#1 v1.5) is vulnerable to padding oracle attacks. To overcome these issues, this study proposes a hybrid cryptographic model which integrates with the key matrix probabilistic scheme (KMPS) using multi-prime RSA architecture and learning with errors (LWE), alongside a modified padding scheme (PKCS #1 v1.5e) utilizing random noises. The feature of this method is the construction of a non-deterministic key matrix combining a random base matrix with a diagonal matrix derived from multi-prime RSA parameters and interjecting Gaussian probabilistic noise. This is followed by security validation using the National Institute of Standards and Technology (NIST) Statistical Test Suite. Experimental results demonstrate that for a 10 x 10 matrix, the ciphertext achieves a P-value of 0.788 in the frequency monobit test, significantly exceeding the NIST threshold of 0.01. This means that the output is statistically indistinguishable from random noise. An avalanche effect of 50.13% is observed, demonstrating strong compliance with the avalanche criterion and indicating substantial resistance to differential cryptanalytic attacks. The internal key entropy is enhanced to 6.28 bits. This model provides a robust solution for securing sensitive database records and transaction logs in enterprise environments, as well as ensuring confidentiality without compromising computational efficiency