The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

will do!

does this PR lack a merge, or is the amount of code changes intentional and really part of this PR only? (+318 - 170)
if intentional, could you explain what it does and why all these changes are necessary?

avoiding masks if they're not necessary has been one line before
attention_mask = attention_mask if not torch.all(text_attention_mask) else None
(on a boolean mask)

@dxqb i intially mis-calculated and was adding support for the pipeline to be compiled, i will revert and simplify

@dxqb i intially mis-calculated and was adding support for the pipeline to be compiled, i will revert and simplify

thanks!
regarding compile,

• Regional compilation is usually as efficient as full compilation (even if it was possible). Regional compilation compiles the transformer blocks, but not the entire pipeline.
• whenever you want to branch depending on GPU data, that's a graph break for torch.compile. Either inefficient, or fails depending on fullgraph. Or, in less abstract terms:
  you want to set your attention mask to None if the entire attention mask is True. But the attention mask lives on GPU. Checking whether all values of a tensor on GPU are True requires a transfer back to CPU - that's always a graph break and cannot be compiled (efficiently).

Therefore, I'd suggest to

• check for an all-True mask in the pipeline
• pass a None-Mask to the transformer block in that case - but don't check any GPU data in the transformer block, so the transformer block can be compiled.

Thanks!

Could you update with the following things?

* Shed light into what caused the speed regression

Here is a benchmark of the impact of using a mask unnecessarily (second graph): #12870 (comment)
torch SDPA falls back to a non-flash algorithm if a mask is used.

yes i also benchmarked the mask and got:

thanks @dxqb please check now

I tested this #12987 on L20, and there was an 11% performance improvement in my test.

And do you find that to resolve your concerns?

I think this has solved my problem. Based on this PR, the performance in my test cases has been working as expected.

thanks @dxqb please check now

I don't have a simple test case, but code looks simple and good now

thanks again @kashif !!