Did you configure a retroarch via the Setup script?
Or did you write your controller-config to the retroarch.cfg?
If you wrote your controller-config to the opt/retropie/configs/all/retroarch.cfg
try the following:
(over ssh from another computer) do
sudo nano /opt/retropie/configs/all/retroarch.cfg
and copy your controller-config.Then do
sudo nano /opt/retropie/configs/n64/retroarch.cfg
and paste the lines to that config-file.
You could also try to set up the retroarch Controller via
sudo ./Retropie-Setup/retropie_setup.sh
and choose the corresponding option.