Entropy Coding for Non-Rectangular Transform Blocks Using Partitioned DCT Dictionaries for AV1

Recent video codecs, e.g. AV1, VVC, apply a Non-rectangular (NR) partitioning to combine prediction signals using a smooth blending around the boundary, followed by a rectangular transform (TX) on the whole block. TX on each NR residual separately is not yet supported. A recent NR TX technique [1] demonstrated promising gains in an experimental setup outside the reference software. This method employs the regular inverse-DCT at the decoder to reconstruct a rectangular signal while discarding the signal outside the region of interest. This design is appealing due to the minimal changes required at the decoder. The method uses a partitioned 2D DCT as a dictionary to find a sparse representation of the NR signal, with scaled representations serving as TX coefficients. These coefficients typically exhibit properties distinct from those of DCT TX coefficients. Therefore, the established entropy coding schemes in video codecs, which are primarily designed for DCT coefficients, are not well-suited for optimally encoding these TX coefficients.