maybe i understand your problem
anyway i don’t have ps3 controller, but ps4 and i at the moment i didn’t get connection over bluetooth with my retropie… :(
but your issue seems that your controller doesn’t work in game, but in menu?
If in menu works, the problem is that for every emulator you need to map your controller, so start your rom, then press ps button and you have access to retropie menu, here you can go settings,device and map it.
I’m going to mind, so the path (menu, settings and device) maybe isn’t right, but anyway try this way IF in retropie menu works