Ok, so I am guessing you are looking in mupen64plus.cfg like here: https://www.youtube.com/watch?v=lh0n5PWN2lI&t=413
See where it says mode = 2, you want to set your mode to 1 or 0.
Mode 1 pulls key settings from InputAutoCfg.ini, and mode 0 uses whatever keys you set in mupen64plus.cfg.
I prefer mode 1 so will recommend setting it to 1 for player 1 and 2, typing the name of your controller as it appears in InputAutoConfig.ini (which I am guessing is ‘Sony Playstation(R)3 Controller’ just like the video), and setting up your keys up in the InputAutoCfg.ini file.
For player 3 and 4, just set the mode to 0 and delete the name, or set it to something else other than the name of your encoder, then only your ps3 controllers will be picked up!
Hope this helps, let me know if not.