Opt-in compliance is a different topic. As the title of this topic implies, the principle motivation for implementing detection keys is improving the user experience of shielded ZEC.
The problem we have with the current Zcash light client protocol is that it’s not particularly lightweight. It requires that the “light” wallet trial-decrypts every shielded transaction (for a given shielded pool) in order to detect shielded (or shielding) transactions. While the use of compact blocks (together with not storing and validating the entire chain) means that the bandwidth and resource requirements are lower than running a full node, it’s still a significant amount of effort. To couch it in terms of boxing weight class terminology, it’s probably closer to middleweight than lightweight. It’s one of the reasons that mobile wallets struggled during the Sandblast phase (and I think we can all agree that was a sub-optimal user experience for most, if not all, Zcash mobile wallet users), and, in the long term, it’s probably not scalable as Zcash achieves mass adoption.
(Note that this is absolutely not intended as a criticism of the Zcash light client protocol or its designers. It was a major and highly successful step forward for Zcash that made it possible to use shielded ZEC on a mobile device while enjoying a very high degree of privacy. The ECC team deserves ongoing respect and kudos for conceiving of, and rolling out a solution that moved us from “You need to run a full node” to “Just install a wallet app on your phone…”.)
In an ideal world, a Zcash light wallet would only need to decrypt its own transactions. Two ways of achieving that are out-of-band note transmission (where the transaction information is transmitted directly to the recipient’s wallet instead of being posted to the blockchain for them to find themselves) or outsourcing the job of payment detection (through trial decryption) to someone else.
Out-of-band note transmission requires some form of messaging solution, which is non-trivial, and I would guess that the implementation/deployment timeline would be significantly longer than that of detection keys. However, as is hopefully clear from the table I posted above, I think it’s something we should definitely work towards in the medium- to long-term.
Outsourcing payment detection involves privacy trade-offs. Right now, you could (in theory) give someone else your incoming viewing key(s) so they can trial decrypt every shielded transaction and tell you which transactions are yours. The privacy trade-off is that they get to see the details of your incoming transactions. That might be acceptable if you trust the person you’re outsourcing to (or if you’re actually “outsourcing” to a service/node that you control) but there’s a better option.
Detection keys would change the privacy trade-off of outsourcing payment detection to a third party from “they get to see the details of my incoming transactions” to “they get to find out which transactions are my incoming transactions” (but don’t learn anything more - i.e. your payment address, the amount being sent to you, and the contents of the shielded memo field remain private). Future cryptographic improvements (e.g. OMR) might even improve that trade-off further.
Additionally, the reduced bandwidth requirements would make it more practicable for the light wallet to interact with the full node over Tor, such that the full node doesn’t even know the light wallet’s IP address.
(Aside: I think that detection keys will also reduce the bandwidth requirements such that the current light client protocol’s privacy trade-offs could be preserved by allowing the light client to trial decrypt using a single transaction field, instead of the whole transaction. Whilst I don’t think that’s where the true win lies, I could imagine it being implemented in lightwalletd as an interim step.)
Anyway, ZF is ready to lean in and support an effort to roll out detection keys by (a) helping design the necessary protocol-level changes (e.g. to the transaction format), and implement them in Zebra, and (b) implementing blockchain scanning using detection keys in Zebra.