Much like raspbmc, OSMC is a full operating system focused around Kodi. You can try to install the retropie_setup skript on a OSMC installation – i think it’s still debianbased – but I’m not sure that it will work.
The option to install Kodi in the retropie_setup is exactly that… it installs the kodi stuff on Raspbian. It’s not a full operating system or anything and you can’t install a different OS through Retropie. You could theoretically check the OSMC sources if available for all the changes made (i.e. the Kodi skin used by OSMC) and pack that together to install it on top of a Kodi installation but that’s a lot of work.