[sselph]

[quote=85550]please consider adding in Vectrex support[/quote]
Adding new systems isn’t too difficult especially then there are only 30ish games. The issue with this platform is that thegamesdb doesn’t contain the platform or the games. If you get them added to the DB I’ll happily get the hashes and IDs added to my mapping.

[quote=85552]RetroPie’s (v2.4x beta, not sure about earlier versions) default MAME emulator is MAME4ALL, which is a fork from MAME v0.37beta5.[/quote]
Thanks. When I get a moment I’ll see if I can do a more accurate job of scraping the mame games for the RetroPie.

[sselph]

Parsing the history file doesn’t look too difficult. Find $info=<name>\n$bio then grab text until I get to a – UPPERCASEWORD – line or an $end. That appears to be the description portion. I’m clueless when it comes to MAME so the data I’m pulling from mamedb says it is 1.47 I would’ve assumed 1.57 was just more complete version of 1.47 but you are saying that some files were renamed. This file I found seems to show the renaming http://www.progettosnaps.net/renameset/pS_renameSET.txt but seems like they are shuffling things around with every version.

What version is retropie supposed to be using? From there I could verify that the files are correctly named based on hashes of the internal files then go through the renaming to get them to 1.47 for mamedb data and to 1.57 for the history.

[sselph]

The data is coming from mamedb.com and there doesn’t seem to be a description. If there is another source that is indexed based on filename that has the description, let me know and I’ll see if I can pull it in.

[sselph]

Thanks!

Some good news. I was able to rewrite the core functionality of my scraper in C++ and get it merged into Emulation Station so it will eventually be an option in the list of scrapers.

While I was there I noticed a pull request to add a mame scraper so I ported that over to Go and added it to my scraper and you can access it with the -mame flag. It should get name, image, player count, rating, developer, genre, date. I don’t have any MAME ROMs so I haven’t fully tested it.

[sselph]

I actually added that over the weekend try v0.5.8-beta and let me know if it is working correctly.

[sselph]

Ah for usb that is automounting you can modify the /etc/usbmount/usbmount.conf and edit the FS_MOUNTOPTIONS to be something like the following if the drive is vfat FS_MOUNTOPTIONS=”-fstype=vfat,gid=users,uid=nobody,umask=002,sync” which will create the mounts so that all users in the users group can modify them. You can reboot or possibly remove and insert the drive again to have the changes take place.

[sselph]

Seems the permissions have gotten in an odd state so the owner is root but pi can read. if you run something like:
sudo chown -R pi:pi /home/pi

it will set the owner of the pi folder and all subfolder back to pi user. If you are saving to a folder not under /home/pi you’d need to specify that folder.

EDIT:
Sorry, didn’t notice that the roms dir isn’t supposed to be owned by pi so might need to do the following to revert that back
sudo chown root:pi /home/pi/RetroPie/roms
sudo chown root:pi /home/pi/RetroPie/roms/*
sudo chmod g+w /home/pi/RetroPie/roms/*
sudo chown -f -R pi:pi /home/pi/RetroPie/roms/*/*

[sselph]

From what I can tell by reading the EmulationStation code, any extra information you add would not be parsed by ES so I don’t think would be available to any theme. It iterates over the expected list of attributes and gets the value from the XML or uses a default value if it doesn’t exist.

[sselph]

I’ve been working on MAME but it has been slow going for a few reasons. We had our first child a few months ago which reduced the amount of free time, I’m not familiar with MAME, the process and datasources are different from what I have now, and there are just so many titles. I plan to add a mame/fba mode that switches from hash matching to name matching and uses a different source of data for the images.

Floob has a video on getting MAME/FBA gamelists built:

[sselph]

-use_cache affects the -use_gdb in that it doesn’t actually download data from the gdb but uses my cached version of the gdb data. -use_cache with -use_gdb=false doesn’t change anything. -use_ovgdb and -use_gdb together will check GDB first then fall back to the OpenVGDB if there isn’t a match in my hashes. So all three would use my cached version of the GDB if I have a matching hash in my DB then fallback to the OpenVGDB if there isn’t a match.

-skip_check
I just noticed some issues with gdb check but how it is supposed to work is if you are using gdb and and not using the cache then I try and determine if the gdb is up first by trying to fetch the game with id=1. I do this so I can give a nice looking error upfront. The -skip_check flag is there in case there are issues with fetching this game but the user knows for sure that the GDB is otherwise working. Right now the GDB check runs even if you aren’t using the GDB, I’ll fix that.

[sselph]

I’m glad it helped. For the DB the simple solution might be to just provide the base xml file that would be appended with missing information. The other option is to have you provide a leveldb or sqlite type db with the hash and information but that might be a lot of overhead.

[sselph]

I originally designed this to build the gamelist.xml on my desktop then copy everything over to the pi so I added options to give the directories on the local and remote systems but the default, to make things easy, was to copy it to the rom dir. It would be possible to add an option to detect if it is running on a retropie installation to be smarter about directories.

The hash not found means that it hashed the rom file and didn’t find a match in the list I compiled. This means the hash wasn’t part of the no-intro set which only has known good rom hashes or I didn’t find the game on thegamesdb. With atari the coverage is not great, I only found ~%50 of the no-intro roms in thegamesdb.

[sselph]

This error is from Go trying to detect the format of one of the images it downloaded and can’t match it to jpeg or png. If the error is happening consistently, maybe there is something odd with the image or thumbnail on thegamesdb, you could try the -use_cache to use my copy of the images. Maybe Google’s caching service will have fixed the issue.

[sselph]

That is unfortunate because it looked like it could be a good alternative. The stuff I looked at was okay but didn’t look at atari. I wanted to start trying to get basic information even if it was not as complete but the last thing I want is something being incorrectly identified. I can filter out results from ovgdb that don’t have description and images which might help. I can also switch -use_ovgdb flag to default to false but in the meantime you can do -use_ovgdb=false to disable it.

[sselph]

Added OpenVGDB as an alternative datasource. It should hopefully fill in some of the gaps in theGamesDB’s DB. It will use this DB when it can’t find data in my original hash.csv lookup. I’ll get back to trying to add MAME now.

[sselph]

There are a few systems you can use zip files with here they are. There are issues with segacd and possibly other systems if the roms are large since the emulator has to hold the extracted rom in memory and the pi doesn’t have a ton of memory.

pi@raspberrypi ~ $ grep -B1 .zip /etc/emulationstation/es_systems.cfg
<path>~/RetroPie/roms/atari2600</path>
<extension>.a26 .A26 .bin .BIN .rom .ROM .zip .ZIP .gz .GZ</extension>
—
<path>~/RetroPie/roms/c64</path>
<extension>.crt .CRT .d64 .D64 .g64 .G64 .t64 .T64 .tap .TAP .x64 .X64 .zip .ZIP</extension>
—
<path>~/RetroPie/roms/fba</path>
<extension>.zip .ZIP .fba .FBA</extension>
—
<path>~/RetroPie/roms/mame</path>
<extension>.zip .ZIP</extension>
—
<path>~/RetroPie/roms/neogeo</path>
<extension>.zip .ZIP .fba .FBA</extension>
—
<path>~/RetroPie/roms/megadrive</path>
<extension>.smd .SMD .bin .BIN .gen .GEN .md .MD .zip .ZIP</extension>
—
<path>~/RetroPie/roms/segacd</path>
<extension>.smd .SMD .bin .BIN .md .MD .zip .ZIP .iso .ISO</extension>
—
<path>~/RetroPie/roms/sega32x</path>
<extension>.32x .32X .smd .SMD .bin .BIN .md .MD .zip .ZIP</extension>

[sselph]

[quote=82648]Thanks, I am checking this out now. Are there more clear instructions somewhere? I downloaded the zip but the directions found on the main page don’t make a lot of sense to me.[/quote]

Sorry my directions are very terse. Floob made a nice video but essentially:

• extract the zip and copy the scraper file to the rom directory you wish to scrape.
• ssh to the pi and cd to the rom directory ie( cd ~/RetroPie/roms/snes/ ).
• chmod +x scraper
• ./scraper -thumb_only
• if you get an error about gamesdb being down do ./scraper -thumb_only -use_cache

[sselph]

I added some Atari 2600 hashes to the mix. The coverage wasn’t as good as I would’ve liked. I’m in the process of refactoring so I can pull in other data sources which will hopefully fill in some of the gaps with some basic data. Sorry it took a little longer than normal, don’t have as much time these days.

[sselph]

The issue these days is usually the fact that https://thegamesdb.net/ is down which is where the scraper is downloading its data. The developer has to manually log in a reboot his system when it happens so there is sometimes a delay. You can try again later today or tomorrow.

I created a scraper and a cache of the DB(-use_cache flag) but it only works for a few systems(NES, SNES, GB, GBC, GBA, MD, TG16, SMS, GG, 32X)
https://github.com/sselph/scraper
https://github.com/sselph/scraper/releases

floob created a nice video on using it:

[sselph]

I thought I had GBA support already but never really tested it so maybe there are issues. I can look into the atari support but was currently looking at what it would take to add MAME but with something like 28000 games it might be easier to add atari.

[sselph]

What is the file extension of the snes roms? The megadrive emulator supports .zip but snes emulator doesn’t so if that is the case you may need to unzip them.

[sselph]

Does sudo apt-get install libgomp1=4.7.2-5+rpi1 help?

That is the version I have on my recently installed 2.3 build.

[sselph]

Could you create your own custom bash script that did something like:

if [ -f […]/snes/$1.cfg ];
then
/opt/retropie/[…] –config [….]/all/retroarch.cfg –appendconfig [….]/snes/retroarch.cfg –appendconfig [….]/snes/$1.cfg $1
else
/opt/retropie/[…] –config [….]/all/retroarch.cfg –appendconfig [….]/snes/retroarch.cfg $1
fi

Then replace the entire command with /home/pi/myscript.sh %ROM%

[sselph]

Odd is this only with the gamelist.xml there? If so, maybe there is something about certain file names or the downloaded data that I’m not escaping correctly and/or ES isn’t probably handling? I’ll try and reconstruct your gamelist.xml from the information you sent and see if I see something that might cause that. I know ES doesn’t like unicode so I may double check and make sure to remove anything that can’t be encoded in acsii.

[sselph]

Yeah turbographx-16 was a low hit rate. Of the hashes I did have ~50% matched and appears the hashes from no-intro aren’t complete. That hash you have doesn’t match the hash I have for that game. If you want you can send me all the sha1 hashes along with the file names for your collection and I can update my DB. shopt -s globstar && shasum **/*.pce from a linux machine will do the trick easy enough or if the tool you are using can do a similar output that’d work. You can send it to me in a PM or email.

[sselph]

This comes from emulationstation the code that does the replace only looks for a single %ROM% instance.

This is the replace call:
https://github.com/Aloshi/EmulationStation/blob/d89ab913abdd90e559e48a38a07fd571cbf8a607/es-app/src/SystemData.cpp#L114

This is the replace code:
https://github.com/Aloshi/EmulationStation/blob/d89ab913abdd90e559e48a38a07fd571cbf8a607/es-app/src/SystemData.cpp#L64

You could send a pull request to alter the replace code to something like the following. I don’t write c++ so not sure if it is 100% correct.

size_t pos = str.find(replace);

while (pos != std::string::npos) {
    str.replace(pos, replace.length(), with.c_str(), with.length());
    pos = str.find(replace, pos + with.length());
}
return str;

[sselph]

Yeah the GBC issues should be much better. I think there was ~60% coverage in theGamesDB. I also just added in your new flag -nested_img_dir to tell it to create the nested sub directories under images and it should create them all for you.

[sselph]

This error is thrown when picodrive can’t determine the media type. Here are some of the possible issues I can think of:

• If the version of picodrive you have doesn’t support the .gen extension. Try renaming one to .bin to test and if that works you might try downloading a fresh 2.3 sd image.
• A less likely option, since you said they work on a friends retropie, is that the files are not a raw binary file and are in the multi game doctor format which isn’t supported in picodrive and you can convert them using a tool like SBWin to a bin.
• The files were corrupted during the copy.

[sselph]

I haven’t played around a ton with the builtin scraper but if I remember correctly there is an option to only scraper things missing images. You might be able to just delete everything in your gamelist.xml and images that is incorrect then manually scrape to get the real data. But the amount of time required inspect everything for issues might be comparable to manually scraping everything.

[sselph]

Haha oops. I forgot to actually push the new CSV file. It should work now. If you’ve run the script in the past 30m just rm /tmp/hash.csv to clear the cached copy of the hashes.

The feedback definitely helps.

[sselph]

While theGamesDB was up today I scrapped all the games my script can match and propped up a service to serve images and xml to mimic the API. Now if the DB is down you can use -use_cache to query my service. To save bandwidth I only downloaded the thumbnail sized images but it is better than nothing.

[sselph]

http://thegamesdb.net/ Which is the server it is connecting to has been having issues for the past couple of weeks and is currently down. You can attempt to go there in a browser to verify it is up before you try scraping again.

[sselph]

I hopefully fixed the GBC entries in the DB. I also added zip support. Right now it is a little dumb in that it searches the files in the zip for the first one with a valid extension and attempts to hash it. It doesn’t look to see if the extension is one that should be zipped (currently only MD).

ceuse, I’ll take a look at separating the images into sub folders, and will make sure the script generates the folder structure for you. It originally did this because of a bug in the parsing of paths in windows so when I fixed the path issues this side effect went away.

[sselph]

Rom seems fine and I see the exact hash in the no-intro set. I must have made some mistake generating my csv. I’ll go back over gbc dataset to figure it out. Thanks for finding the issue.

[sselph]

Sure gameboy color roms are a simple raw binary format so you can do shasum *.gbc and get a list of hashes to file names. Feel free to send me the list in a file. If you want to troubleshoot you can look at the csv here:
https://stevenselph.appspot.com/csv/hash.csv

I can think of one issue. If these games were clones of a normal gameboy game just with added color they could be listed in thegamedb as gameboy and not gameboy color. I might just need to expand my search.

[Author]

Posts