There is no problem whatsoever with XenForo permissions. ZERO. The only issue is your understanding of them. They are laid perfectly ONCE YOU UNDERSTAND THEM. You can give up, or you can try again. If you stick with it, you will experience that "AHHHHH" moment, and everything should be fine from that point.
The one piece of advice I will offer for now....Never use the Never permission setting. I have not used it one time for anything. I have zero permission issues. I have all my forum areas properly restricted so the right users have access and no one else. There are instances where Never is helpful, but for now just don't use it for anything.