The network difficulty would be exactly the same if miners would switch over to other pools. If we would consider a situation where a large portion of the miners switch to another pool and the total hashrate would be the same (that is: no increase or decrease of miners to zcash) THEN the expected payout per miner would also be identical. HOWEVER there is a larger fluctuation on this expected payout per miner depending on the pool's relative hashrate compared to the global hashrate.
The problem with this 50%+ hashrate of flypool may be a reason why poloniex and other exchanges stopped their deposit and withdrawal of zcash. These exchanges still need to make an official statement however, so I have no proof that this is the case. Due to this centralisation of mining power, exchanges can no longer be confident that deposits are actually valid. Increasing the number of confirmations for exchanges is not a solution when flypool has more than 50% of the mining hashrate, see https://en.bitcoin.it/wiki/Majority_attack. The probability for flypool to double spend would in the case of a 50%+ global hashrate share be a 100% success. If they would abuse this power and double spend, it could seriously damage all running operations of exchanges that accepts any transaction from the zcash network.
There is nothing wrong with zcash and its implementation, however due to flypool nobody can now be sure that the transactions in the network are actually valid. There is no way for users to see a double spend in the zcash network, since protected transactions can have their input from void. Eventhough users can check if the total amount of transparent ZEC does not surpass the total mined ZEC, it would still give attackers the possibility to double spend the total amount of protected ZEC in the whole network (at this moment about 63489ZEC). Nodes however would identify a false zero-knowledge proof, but cannot act on it because the blockchain that is the longest is the valid chain in their eyes. In this case the blockchain would be operated by the attackers.
Nothing to worry however, when miners decide to eventually switch their hashpower to other pools the probability of a successful double spend would decrease and the network goes back to normal operating mode. Even in the case where flypool cheated, there would be a chain correction and the chain in which a double spend occurred would be declared invalid. But this may be very inconvenient for zcash users as their passed transactions may be declared invalid.
The fact that flypool has the possibility to inflict damage by having the possibility to do a double spend makes zcash not trustworthy at the moment and therefore I strongly encourage miners on flypool to reconsider and move towards other pools.