I was able to eventually figure this out on my own. Maybe someone else will find this useful, so I’ll go through everything I went through to figure this out. I’m sure I could have made it easier on myself, but online searches kept pointing me to information that was outdated. Anyways, here you go:

First off, I previously tried screwing around with /opt/retropie/configs/n64/retroarch.cfg. Thankfully I had a backup and just restored it to the original. Secondly, I hooked up a keyboard and regained the ability to at least choose the n64 default emulator and specific choices for individual ROMs. So from there I found a n64 compatibility list and started selecting emulators from that sheet.

If I had a game based on lr-mupen64plus (Libretto? I think it’s called sometimes), I started a game with that emulator choice and was able to pull up an on-screen menu via F1 on my keyboard to do a custom mapping for my Xbox 360 controller.

However, most of the other games I was trying used mupen64plus, and either there is no onscreen menu to do the mapping or I can’t find one. Either way, I had to edit the /opt/retropie/configs/n64/mupen64plus.cfg file to make the specific mappings for my controller work.

To even know what the mappings were I had to plug in a keyboard, exit emulationstation and run the following command: jstest /dev/input/js0. It brough up text listing axis and buttons for the controller. I pressed every button, one at a time, watched what values changed for 0, and then made a notation. This is what I came up with for my Xbox 360 controller connected wirelessly to a 3rd party USB to wireless adapter (YMMV of course):

L Analog up:          axis -1
L Analog down       axis +1
L Analog left          axis -0
L Analog right       axis +0

R Analog up          axis -3
R Analog down     axis +3
R Analog left          axis -2
R Analog right        axis +2

Digital up               axis -5
Digital down          axis +5
Digital left               axis -4
Digital right             axis +4

Select    button 8
Start       button 9
Xbox button     button 10

A          button 0
B          button 1
X          button 2
Y          button 3

L trigger          button 6
R trigger          button 7
L shoulder button     button 4
R shoulder button     button 5

I then modified the mupen64plus.cfg file and looked for the [Input-SDL-Control1] section to modify the settings for the controller. In that section I HAD TO set mode = 0 (fully manual), otherwise it would just revert my custom mapping. Right under the # Digital button configuration mappings section is where I put in the customizations. Obviously these are my preferences but maybe this will be an example for someone:

# Digital button configuration mappings
DPad R = "hat(0 Right)"
DPad L = "hat(0 Left)"
DPad D = "hat(0 Down)"
DPad U = "hat(0 Up)"
Start = "button(9)"
Z Trig = "button(5)"
B Button = "button(1)"
A Button = "button(0)"
C Button R = "axis(2+)"
C Button L = "axis(2-)"
C Button D = "axis(3+)"
C Button U = "axis(3-)"
R Trig = "button(7)"
L Trig = "button(6)"

Lastly, it bothered me like crazy that there wasn’t a hotkey for me to exit the mupen64plus emulator from my controller. Since I did the hard work earlier to understand the configuration of every button and stick on my Xbox 360 controller, I just had to pick a single button for the hotkey (I believe that key combinations for mupen64plus are not allowed, but maybe I’m wrong). I decided to use the Xbox button, which for my controller was button 10. From looking online, I had list the joystick number. Since it tested as js0, then it’s joystick 0, or J0 for the parameter, and B10 for “button 10”. I modified the following area to allow the hotkey:
Joy Mapping Stop = "J0B10"

Frustrations with the controller on mupen64plus are now over! Hope this helped someone other than myself.