Thanks to EBZero for his post at the RetroPie subreddit for pointing me in the right direction (https://www.reddit.com/r/RetroPie/comments/3p5pkq/beware_using_select_as_a_joypad_hotkey_in_nes/).
The problem arises because the Select key is used in Star Tropics to navigate down in the game menus as well as being the hotkey. The easiest way to get around this is to remap the Select key to another button.
1. Launch Star Tropics from ES and before it loads, press any key or js0 to bring up the runcommand menu, then select option 9 (Edit custom RetroArch config for this rom).
2. Add the following line to the custom config file:
input_player1_select_btn = "x"
where x is the number of the button you wish to use for Select.
3. Now exit and launch the rom. You should now be able to enter a name and start a game using the remapped Select button.
This works in lr-fceumm but not in lr-nestopia. I haven’t tested Star Tropics 2 but I assume it should work with the same method.