A Fuzzy Large TabNet-Based Model and Its Distillation Learning for Noisy-Labeled Data via Consequent Additive Decomposition

In this study, in order to exploit an interpretable fuzzy large model on clean dataset from large and/or complex noisy-labeled data and then generate its student model appropriately through knowledge distillation for its lightweight usability, anInterpretablefuzzylargeTabNet-basedmodel, called IFLTNM, based on the well-proven deep neural network TabNet and interpretable fuzzy rules is first proposed as a type of fuzzy large models. By means of both the well-established memorization effect and the proposed consequent additive decomposition, after interpretable antecedents are fixed, the consequent parameters of all fuzzy rules in the IFLTNM model are appropriately determined at early training stage for almost clean training subset from the whole noisy-labeled training set. After that, by regularizing the learning objective of the proposed knowledge distillation on almost clean training subset with an extra loss function on almost noisy-labeled training subset obtained after the IFLTNM's training, the proposed learning objective is optimized to have IFLTNM's student model s-IFLTNM consisting of the same number of fuzzy rules sharing the same interpretable antecedents as in IFLTNM yet having linear consequents. IFLTNM has its structural and training novelty in the sense of leveraging both the memorization effect and consequent additive decomposition to train a fuzzy large model and acquire almost noisy-labeled training samples, and s-IFLTNM is distilled from IFLTNM creatively through leveraging both almost clean and noisy-labeled training samples instead of only clean samples. Experimental results on real-world benchmark datasets demonstrate the effectiveness of IFLTNM over the state-of-the-art methods,e.g., achieving 84.94% average testing accuracy yet preserving interpretability for thecovertypedataset even with 30% symmetric label noise, and the power of s- IFLTNM in the sense of both a significant reduction of computational burden and model complexity of less than 400 on the adopted datasets with only a marginal drop in performance.