I feel your pain :)

When you install the xbox 360 driver it writes to /etc/rc.local

The initial setup that comes default is option 2 on this page
https://github.com/retropie/RetroPie-Setup/wiki/Setting-up-the-XBox360-controller

Specifically it is set to look for only 2 controllers. When you attempt to load controller 3 and 4 they’ll go black because it is only looking for 2 total controllers.

Edit /etc/rc.local and remove that default single line setup. I believe that should fix your issue.