The only limit on the size of a transaction is that it must fit in a block, which is limited to 2 MB. There was a 100 KB transaction size limit at launch but it was removed in the Sapling network upgrade:
Speaking for myself: I think fee policy cannot be a particularly effective anti-DoS measure. Fees would have to be too high (for normal usage) in order to work for that purpose. Fee policy could potentially help to some extent to buffer spikes in demand due to organic usage. For now, I think that the changes in v5.1.0 are likely to be sufficient to address any short-term performance problems with current transaction load.
Edit: no, the changes in v5.2.0 are needed as well to address the scan performance issue.
[Edit 2023-12-14: I was wrong about this, as ZIP 317 proved. I had changed my opinion between this post and a month later, around 2022-07-22 when Kris Nuttycombe proposed a variable-fee approach based on the sum of the numbers of shielded inputs and outputs.]