Scraper not finding some games

[cyborguk]

Hello,

Fully appreciate the games db can’t have all games, however can I manually update games that have no images?

[herbfargus]

You can submit missing images to thegamesdb so others can benefit if the gameadb doesn’t have an image.

You can also transfer over your image and then press shift (or select, I can’t remember) once you’ve navigated to the game and edit the image path manually to where you transferred your image over. Just note that you’ll have to select quit from the emulationstation start menu If you want those manual changes saved.

Otherwise you can manually edit the gamelist.xml for that system which accomplished the same thing.

[cyborguk]

Where is the gamelist.xml held, can’t locate it? I was hoping to copy the images to the default location to test first.

If all looks OK I can upload to the gamesdb, I will Google their site.

[sselph]

If you used the builtin scraper or my scraper via the retropie-setup script the gamelist is in ~/.emulationstation/gamelists/(system).

If you used my command line scraper it is in the rom folder.

[nosedeath]

sselph’s scraper is AMAZING!! Use it. so much quicker. It had just about all of them too.

Thanks sselph!

[cyborguk]

I’m checking this out tonight!!

Thanks, I have lots of C64 games to do.

[sselph]

I have an issue open to add C64 support and just haven’t had the time to learn about the file formats involved.

[cyborguk]

Oh ok, would be nice to see a scraper take hold of the Amiga, C64 and Spectrum one day.

[cyborguk]

Sorry where in retropie-setup script do I install this?

[chito]

I dont understand how there arent complete gamelists for retropie with all the meta data and images. complete packs… sselph’s tool matches roms to data by Hash… if a tool like his had its own complete db with images and data specifically for ES we would be set. I get that this is the idea behind scrapping these sites but im saying lets remove that dependancy. even have a community driven open contribution wiki or something.

[redhalo]

Well, there isn’t complete info for all games. These sites being scraped are still constantly being updated. A “community driven open contribution wiki” is exactly what these sites are. The solution you describe is exactly what’s in place.

[herbfargus]

I think it’s better to have a scraper than complete packs as a lot images can take up space, so why add images you don’t need?

[chito]

we really should make a solution specific to retropie/es. scrapping is a messy slow solutions and less than ideal should the site change the scraper will break. Also we could add support to select different images packs (screen shoot, title, poster, cartridge, boxart) and added easily.

[chito]

or maybe thegamesdb can create a public API… otherwise we can create our own data base of meta data. and that database would only need to scrap data (data mine) off these sites maybe a few times a year.

[chito]

[quote=109129]I think it’s better to have a scraper than complete packs as a lot images can take up space, so why add images you don’t need?
[/quote]

the solution im suggesting would use a complete pack and then another tool like sselph’s could customize your gamelists for what it finds in your folders and only transfer or use what is relevant

[redhalo]

The “complete pack” is thegamesdb, the scraper is what we have now. You are literally describing the current use case.

[chito]

[quote=109134]The “complete pack” is thegamesdb, the scraper is what we have now. You are literally describing the current use case.
[/quote]

actually its not, switching from a scrapping solution to an api is wayyyyyyyy different and way more efficient, faster and customizable

[redhalo]

http://wiki.thegamesdb.net/index.php/API_Introduction

thegamesdb.net has an api, sselph’s scraper uses the api.

[chito]

I see, well thats a relief lol, so why is it called a scrapper? its not scrapping.

I hope they add support for the api to return your choice of game artwork (boxart, screenshot, title etc)

[sselph]

[quote=109118]Sorry where in retropie-setup script do I install this?[/quote]

Sorry I haven’t actually used this from the menu to see but I’ve heard people say the experimental area. Should say something like “Scraper for EmulationStation by Steven Selph”. Herbfargus would know.

[quote=109135]actually its not, switching from a scrapping solution to an api is wayyyyyyyy different and way more efficient, faster and customizable[/quote]

My scraper(for console roms) maps the contents of the rom(via hashing) with the ID in thegamesdb to query the API. It then formats the API response in the format suitable for the XML. This is a fairly efficient method for the systems I have built the mappings for since it only downloads what is needed and doesn’t perform any unneeded search API calls.

There is a project that is attempting to do something like you mention. It relies on the file naming to be done according to the no-intro naming. They used my scraper to seed much of the information.
https://github.com/digitalLumberjack/recalbox-gamelists

[chito]

How hard is it to add a new system like C64 for cyborguk? I personally would love to have Neogeo pocket. I was looking at creating my xml manually :(

Using the hash is way better cause i use no-intro and datomatic to create 1 game 1 rom sets. so the matching of hash to id you would’ve had to have done that manually? or did you create a tool that tried to find a match and you confirm (kinda like the way ES scraper works) either way thats a huge amount of work really appreciate it sselph.

for arcade games (in particular mame) could you use the mameinfo dat files to create the xml gamelists? images could still be retrieved from thegamesdb

[sselph]

[quote=109140]so why is it called a scrapper? its not scrapping.[/quote]
That is what Aloshi called it in ES so I just used the same terminology. It does actually scrape HTML for MAME info.

[quote=109140]I hope they add support for the api to return your choice of game artwork (boxart, screenshot, title etc)[/quote]
The API returns all the images that are available. boxart is almost always there where screenshot, etc are less common. My script, if run from the command line, you can accept the flags -mame_img or -gdb_img depending on which system type you are processing. You can see how to use these with -help

[quote=109144]How hard is it to add a new system…So the matching of hash to id you would’ve had to have done that manually? or did you create a tool[/quote]
To add a new system I have to find a set of hashes for the rom set. Typically from no-intro. I have to determine how the rom data is formatted. Some systems like nes have headers which aren’t included in the hash. Some systems like megadrive and n64 have multiple encodings for the same data so I have to convert them on the fly to the correct format. Once I’ve determined how to hash a file, I started mapping hash to ID with the help of some scripts and spreadsheets. So it has some automation but still very manual.

[quote=109144]for arcade games (in particular mame) could you use the mameinfo dat files to create the xml gamelists?[/quote]
For arcade platforms I am not doing the best job. I use the name since that is a good key for these systems then I scrape mamedb.com/game/(name) plus some info from a history file. I should try and map the name to the correct name for the mame version, but what I have has been good enough for most people.

[chito]

[quote=109145]I started mapping hash to ID with the help of some scripts and spreadsheets. So it has some automation but still very manual.[/quote]

I tip my hat to you. Thats heaps of work. Well Im very keen to complete the NGP “scrapping”. I was going to edit my gamelist.xml manually but happy to help add this system to your tool if you like.

[cyborguk]

Sselph I will look again tonight, but I did look there and couldn’t see it, it was at the end of a long day at work so my eyes were not the best.

[herbfargus]

It’s a hidden folder so you may have to make hidden folders visible to see the gamelist xmls.

Also just for reference:

https://github.com/RetroPie/RetroPie-Setup/wiki/scraper

[cyborguk]

I managed to install from the experimental menu and it also asked which emulator to search and scrape, which I did for the NeoGeo and PSX, but not images show up, it did scan far too quickly too, did I miss a step?

[Author]

Posts