It is a driver issue with the retrousb adapters, I have the same issue. A change was made to the kernel that messed up the drivers a while back. I have mostly solved it by rolling back the kernel to a previous version that worked with the adapters.

sudo rpi-update 4e2845e0cbe7e701f0431e66a26f730de264e7a6

enter the above through putty or terminal and it will take care of the problem for the most part. If I ssh into my pi and then return to playing games it will still scroll sometimes but a full power off corrects it. Only work around I have found. The problem does not happen with the iBuffalo controllers.