NextPosFormer: End-to-End Next-Positive-Token Prediction with Decoder-Only Transformer

Abstract We present NextPosFormer—an innovative end-to-end generative recommendation framework that, for the first time, achieves unified tokenization of all heterogeneous information sources in recommender systems at the architectural level. These sources include user attributes, item features, store metadata, and diverse interaction behaviors, all seamlessly integrated into a multi-task generative recommendation paradigm. Our core contributions are fivefold: ● Info-Profile Standardized Repository: Multi-domain entities—users, items, and stores—are abstracted into a unified Info entity schema. Structural heterogeneity is entirely eliminated through cross-domain feature alignment and default-value padding. ● Universal Tokenizer: A shared deep encoder seamlessly processes any (info_profile, InteractType) pair—whether user profiles, items, stores, or generalized interaction types (e.g., click, purchase, follow)—mapping them into homogeneous d_model-dimensional tokens, thereby realizing the principle of “Everything is Tokenizable.” ● Interaction-Type-Aware Multi-Task Masking: Configurable binary masks (InteractType_Mask) dynamically suppress task-irrelevant interactions during loss computation. Gradients are updated only for target positive actions (e.g., clicks for CTR or purchases for CVR), while the full behavioral sequence—including negative samples—is leveraged to learn more robust user representations. ● Generative Multi-Task Recommendation Architecture: Building upon the above masking mechanism, we propose a novel generative multi-task recommendation architecture that enables different tasks to focus on their respective positive interaction targets within a unified autoregressive generation framework, achieving coherent multi-objective prediction. ● True End-to-End Joint Optimization: The Universal Tokenizer, causal decoder-only Transformer, and task-specific prediction towers are trained jointly from scratch, with no frozen components or pipeline fragmentation, ensuring globally coordinated parameter optimization. By design, NextPosFormer fundamentally addresses two long-standing challenges in recommender systems: heterogeneous feature fusion and gradient interference in multi-task learning. More importantly, by elevating the input representation from traditional “item ID sequences” to holistic “user interaction sequences,” it provides a standardized interface for native integration of Large Language Models (LLMs) into recommendation systems and opens a new pathway toward discovering Scaling Laws in recommender systems. This manuscript presents the complete model architecture and design principles. Large-scale empirical validation is currently underway and will be reported in a follow-up publication. Keywords: Recommender Systems; Heterogeneous Feature Unification; Universal Tokenizer; End-to-End Learning; Multi-Task Learning; Decoder-Only Transformer; Interaction-Type Masking; Scaling Law