My understanding of difficulty is it sets the rate of how many blocks are created. The more hash power put toward mining at the current block rate, the more the difficulty increases.
Although it’s a brilliant way to control the block creation rate, it creates an environment where only those with the most resources can mine blocks. As they invest in more hashpower to compete with others, the difficulty increases. This slowly weeds out those with less hashpower until they drop out. The number of nodes decrease and the network moves toward centralization.
What if there was a way to have a node difficulty. Average the network solution rate by the number of nodes. If a node sol/s increases over the average network solution the difficulty for that node increases.
The network needs to be responsive and process transactions quickly, but most of the processing power is going to overcome the difficulty and not to generate new blocks.