I have not done this with XBMC, but it seems pretty straight forward. In ~/RetroPie/roms create a directory for applications: mkdir ~/RetroPie/roms/apps
Then, to edit the emulation system config to add an apps section: nano ~/.emulationstation/es_systems.cfg and add:
DESCNAME=Applications
NAME=apps
PATH=/home/RetroPie/roms/apps
EXTENSION=.sh
COMMAND bash %ROM%
The use ctrl-O, ctrl -X to save and exit.
Now, we add the scripts to launch our apps. To create a test: nano ~/RetroPie/roms/apps/test.sh and type:
#!/bin/bash
nano
Then ctrl-O ctrl-X to exit and save.
Load up emulation station, and you should have an apps menu, with our test, and it should launch the nano test editor when selected. For other apps (XBMC, etc) create a .sh file in the roms/apps directory use the command to launch the application in place of nano from the test.sh file.