Method
In this work, we focus on accelerating diffusion-based super-resolution, which is particularly expensive due to the larger size of the input and output, but crucial to video generation pipelines. Most prior works speed up diffusion transformers by reducing the number of steps or modify the attention mechanism. W We instead find that we can skip computation altogether for certain `simple' patches, accelerating computation significantly. We identify these patches with a lightweight CNN that predicts a binary mask, which is then used to route these patches entirely around the transformer. The un-skipped patches that are refined retain their relative positions through a mask-aware rotary positional encoding. Extensive experiments and ablations demonstrate that our method preserves visual quality while significantly accelerating end-to-end generation time. Our method is illustrated in the figure below, and several examples, along with the predicted mask, are visualized at the bottom of the page.
