Add option for selective op AC to filter mm shapes based on fqn

[soulitzer]

Also see discussion in #1372

This PR:

• Adds new config for SAC with the default such that per-op SAC automatically skips all mms with args[1].shape matching that of the Linear at fqn "moe.router.gate"
• Adds general flop/act-mem/correctness tests for AC as well as the new config

[tianyu-l]

This is a great idea!
Since this feature is advanced, could you also help test if the behavior is expected?

It seems this feature does not require distributed, so maybe we can add a unit test file in
https://github.com/pytorch/torchtitan/tree/main/tests/unit_tests

But if it doesn't make sense, feel free to do it in the way you prefer.

[tianyu-l]

seems good enough to

Suggested change

	default_factory=lambda: []
	default_factory=list

or we can default to ["moe.router.gate"] so that we don't need to define it in a lot of tomls.

O/w could you please also update the tomls in https://github.com/pytorch/torchtitan/tree/main/torchtitan/experiments/llama4/train_configs
and
https://github.com/pytorch/torchtitan/tree/main/torchtitan/models/deepseek_v3/train_configs

[tianyu-l]

Note that in float8, we also filter by fqns, in which we are doing reversely
https://github.com/pytorch/torchtitan/blob/main/torchtitan/components/quantization/utils.py#L25
I think one reason could be that the filter over there is applied to the whole model, so one fqn can help map to multiple layers / modules.

I think for AC there's not that much difference between the two. The benefit of doing it the other way may be users don't need to specify accurately the full relative fqn within the AC region. E.g. "router.gate" would also work.

I don't have a strong preference, but maybe let's be consistent with float8 if you don't have strong preference either.

[tianyu-l]

I wonder if we should prefer a shorter name over how accurate its meaning is. How about per_op_sac_filter_fqns? Most users shouldn't really care about the details of implementation; if some users do, they can check the helper message and implementation.

[soulitzer]

Good question. To help reduce the cognitive load of parsing the config file for the average user who I agree won't care about the impl, would it help if the default already include any moe router fqns for TorchTitan models per your other suggestion? This means most configs won't need to contain it at all, so most users won't see it and the advanced users using it will still benefit from a more explicit name.

I think this is consistent with most users already not being aware of what the per-op sac policy is at all, although we could potentially refactor things such that we have specific policies like policy="compute_intensive_excluding_every_other_matmul"