Ok, sorted this out. So for those of you struggling with it or contemplating using a trackball heres how to get it working reliably. Its all down to the config file for advanceMAME in the hidden .advance directory : /home/pi/.advance/
Pick the config file for the version your using (or edit both)
device_mouse raw
device_raw_mousedev[0] /dev/input/mouse0 (or whatever your trackball is assigned to – mine was mouse1)
device_raw_mousedev[1] auto
device_raw_mousedev[2] auto
device_raw_mousedev[3] auto
device_raw_mousetype[0] ps2
device_raw_mousetype[1] pnp
device_raw_mousetype[2] pnp
device_raw_mousetype[3] pnp