[sselph]

That is an odd error. You are getting the error from downloading the hash or later? Can you me a send me a PM with the output of the script? That error means that a request you sent to web server was too large. This usually only happens if you are uploading a file which my script doesn’t do.

[sselph]

Ah retropie doesn’t place the binary in the system path so you can’t just issue scraper but would have to give the full path. Unfortunately, I can’t figure out from the code where it places the binary so you might be better off just downloading a copy by following my instructions here:
https://github.com/sselph/scraper#install-from-my-binaries

Just remember you have to find the latest release value here:
https://github.com/sselph/scraper/releases

[sselph]

Doesn’t seem to work isn’t very descriptive, can you provide more information?

The scraper works for multi-disc games but doesn’t work for iso files since redump hashes are for bin/cue.

[sselph]

The default is to use the no intro name which includes the region. You can do -use_nointro_name=false to use the name from thegamesdb which doesn’t include a region.

[sselph]

[quote=118920]Great work Sselph. One question are there higher resolution pictures available, I use the nbba theme so images are large on the right but in mame the quality is very low, I used the thumbs only switch, should I delete the images and try without the thumbs switch?[/quote]

This is because the images at mamedb are low resolution.

[quote=118528]I want to improve the hash list with some psone game of my collection. Do you use the .cue hash or the first .bin of each game ?[/quote]

The .cue and the main(largest) .bin should work. I used to do all the bins but there are some duplicates across games so the main bin file should work.

[sselph]

Yes -add_not_found -download_images=false and if the scraper doesn’t know about the extension involved you can do -extra_ext=".new" Depending on the name of your files you may need to do -image_suffix="" to tell it to name the images rom.jpg instead of rom-image.jpg

[sselph]

I’m assuming you were scraping with a scraper outside of EmulationStation. One common issue is you have to exit ES. With the recent updates it makes it difficult but when ES exits it writes a new gamlist.xml with the latest last played, play count information. This has the effect of overwriting all the work you may have done.

[sselph]

I’ve been bad about monitoring the forum. I do my best not to create a false positive. The worst case is that the rom isn’t scraped. In those cases you can use the builtin scraper with the option to only scrape roms with a missing image. this will scrape the roms manually that my scraper couldn’t identify.

I created the script after manually scraping then realizing I needed to update retropie and had to do it all again. I figured I would save myself from having to do it a third time :) and the last thing i wanted was something labeled incorrectly.

[sselph]

You can use the -missing missing.csv that will write all files that weren’t scraped to a file with an explanation. Either the hash was not in no-intro, the game matched but was missing in thegamesdb when I did the last matching, or the game is in thegamesdb but no image was found. You can open that file and add a column with thegamesdb ID and I can add those to the master csv.

If you think data in the existing master csv could be helpful you can download it from:
https://storage.googleapis.com/stevenselph.appspot.com/hash.csv.gz

[sselph]

Hi zigurana,

Yeah this is most likely what is happening.

As for adding my code to ES, I have submitted https://github.com/Aloshi/EmulationStation/pull/355 to ES and it is in their unstable branch. This code is somewhat out of date but it is there. Now that Go can create C modules it might be possible to refactor my code or create a small wrapper to allow ES to import it as an optional library so that it is kept more up to date. If you are a programmer and want to look in to this feel free and I’ll be happy to answer any questions.

The other solution to this is that ES was moving to using sqlite for storage and I already have some unchecked in code to also use sqlite. When this happens ES would no longer be dumping everything to the xml on exit which would probably solve the issue.

[sselph]

I just saw another message where someone said the gamelist was placed in their home directory. Check to see if that is the case, if so that would be strange and I’ll have to spend some time diagnosing that.

[sselph]

Hmm the gamelist should be in the same directory where you ran the script was run. I’ve heard some other complaints about this so maybe something has changed.

Anyway if you ran the script from inside a folder with a bunch of roms and didn’t change any of the flags, all the paths should be correct just put the gamelist in the rom folder along with all the roms and the images folder.

[sselph]

Hi cyborguk,

The gamelist.xml is placed in the current working directory. This is the same directory where it is looking for roms and placing images so if those are working it should be fine. You can try adding -output_file /home/pi/RetroPie/roms/(rom_dir)/gamelist.xml to make it explicit. Permissions don’t seem to be the issue since it is able to create an images folder and write to it and it should give an error in that case. The other option is the scraper didn’t find any hashes which doesn’t appear to be the case since it downloaded images. I haven’t changed anything in my script in over a month except adding and removing some hashes in my database.

[sselph]

[quote=109140]so why is it called a scrapper? its not scrapping.[/quote]
That is what Aloshi called it in ES so I just used the same terminology. It does actually scrape HTML for MAME info.

[quote=109140]I hope they add support for the api to return your choice of game artwork (boxart, screenshot, title etc)[/quote]
The API returns all the images that are available. boxart is almost always there where screenshot, etc are less common. My script, if run from the command line, you can accept the flags -mame_img or -gdb_img depending on which system type you are processing. You can see how to use these with -help

[quote=109144]How hard is it to add a new system…So the matching of hash to id you would’ve had to have done that manually? or did you create a tool[/quote]
To add a new system I have to find a set of hashes for the rom set. Typically from no-intro. I have to determine how the rom data is formatted. Some systems like nes have headers which aren’t included in the hash. Some systems like megadrive and n64 have multiple encodings for the same data so I have to convert them on the fly to the correct format. Once I’ve determined how to hash a file, I started mapping hash to ID with the help of some scripts and spreadsheets. So it has some automation but still very manual.

[quote=109144]for arcade games (in particular mame) could you use the mameinfo dat files to create the xml gamelists?[/quote]
For arcade platforms I am not doing the best job. I use the name since that is a good key for these systems then I scrape mamedb.com/game/(name) plus some info from a history file. I should try and map the name to the correct name for the mame version, but what I have has been good enough for most people.

[sselph]

[quote=109118]Sorry where in retropie-setup script do I install this?[/quote]

Sorry I haven’t actually used this from the menu to see but I’ve heard people say the experimental area. Should say something like “Scraper for EmulationStation by Steven Selph”. Herbfargus would know.

[quote=109135]actually its not, switching from a scrapping solution to an api is wayyyyyyyy different and way more efficient, faster and customizable[/quote]

My scraper(for console roms) maps the contents of the rom(via hashing) with the ID in thegamesdb to query the API. It then formats the API response in the format suitable for the XML. This is a fairly efficient method for the systems I have built the mappings for since it only downloads what is needed and doesn’t perform any unneeded search API calls.

There is a project that is attempting to do something like you mention. It relies on the file naming to be done according to the no-intro naming. They used my scraper to seed much of the information.
https://github.com/digitalLumberjack/recalbox-gamelists

[sselph]

I have an issue open to add C64 support and just haven’t had the time to learn about the file formats involved.

[sselph]

Hmm I apologize, I thought I remember there being a platform_id type tag in the xml that told ES the platform but it doesn’t appear to do that. It instead pulls from this file https://github.com/Aloshi/EmulationStation/blob/646bede3d9ec0acf0ae378415edac136774a66c5/es-app/src/PlatformId.cpp

[sselph]

I don’t have the es_systems.cfg in front of me, but if you look at other systems you’ll notice a platform id, like snes is 6, If you add something similar for Vectrex that should help. The ID for Vectrex is 4939.

[sselph]

If you used the builtin scraper or my scraper via the retropie-setup script the gamelist is in ~/.emulationstation/gamelists/(system).

If you used my command line scraper it is in the rom folder.

[sselph]

The current version of emulationstation looks for the gamelist.xml file first in the rom folder and if it doesn’t exist it looks in the ~/.emulationstation/gamelists/(system) folder and finally /etc/emulationstation/gamelists/(system). The images can be anywhere since the gamelist tells ES where to look.

I prefer the gamelist and images in the rom folder since it makes it easier to copy files to and from the retropie. I actually wrote the scraper to run on my desktop and then I copied the files over to make upgrading systems easier.

ES’s author, Aloshi, has the opposite opinion and prefers that the gamelist and images not clutter up the rom folder so the internal scraper places things in the ~/.emulationstation folder.

My script can be invoked with command line flags to tell it where to find the roms and where you will be placing the gamelist and images. This is how it works when my script is invoked from the retropie-setup script. But keep in mind the gamelist can only be in one of three places if you expect ES to find it.

Something like this(with (system) replaced by the system) but you need to double check the paths since I’m doing this from memory. The image path is specified twice since this script was originally designed to be run from a different system, this allowed the host and target system to have a different folder location.

scraper -output_file ~/.emulationstation/gamelists/(system)/gamelist.xml -img_dir ~/.emulationstation/downloaded_images/(system) -img_path ~/.emulationstation/downloaded_images/(system) -rom_dir ~/retropie/roms/(system)

[sselph]

Not in the current version. I think you can use the built in scraper that has an option to only scrape games that don’t have an image. And that has an option to tweak the search if something isn’t found.

[sselph]

Hi sorry if this has already been figured out but you can do this with my scraper but it isn’t that obvious and was broken until just now:

scraper -hash_file=/dev/null -add_not_found -image_suffix=""

If you aren’t on a linux system you can replace /dev/null with an empty file. This tells the scraper to run without a DB so it won’t find anything. Then you are telling it to add things it doesn’t find. If the scraper detects the image is already there it adds it. The default image name is rom-image.jpg so if you want to omit the -image you can change the script’s suffix to be nothing. So you get an entry like this

<game id="" source="">
  <path>./Rom Name.nes</path>
  <name>Rom Name</name>
  <desc></desc>
  <image>./images/Rom Name.jpg</image>
  <releasedate></releasedate>
  <developer></developer>
  <publisher></publisher>
  <genre></genre>
</game>

[sselph]

You can use -use_nointro_name=false to use what is listed in thegamesdb which would omit the region. Another way, as robertybob mentioned, is to post-process the file something like the following to find things in parens and remove them. I haven’t fully tested this but I think it will work.

cat gamelist.xml | sed -e 's/<name>\([^()]*\?\)\( (.*)\)\+<\/name>/<name>\1<\/name>/' > gamelist-new.xml

[sselph]

I make several prebuilt binaries available at https://github.com/sselph/scraper/releases

or if your the type that likes compiling it yourself, there are no special instructions for doing it on windows.

[sselph]

If you are using my scraper you can use the flags -gdb_img or -mame_img depending on what you are scraping

-gdb_img is for console games and defaults to just boxart but you can change to “s,b” to use the screenshot then fall back to boxart if there isn’t a screenshot

-mame_img is for mame and already defaults to the screenshot first.

you can use -help to see the other options

[sselph]

You can use the -use_nointro_name=false to use the names in the games db which shouldn’t have the region. Or you could strip them out of the xml file.

cd ~/RetroPie/roms/nes
mv gamelist.xml gamelist.xml.bak
cat gamelist.xml.bak | sed -e 's/<name>\(.\+\?\) (.\+\?)<\/name>/<name>\1<\/name>/' > gamelist.xml

[sselph]

You can use good tools to get only good roms out and move them to a directory sorted by region. I forget the command flags but there is a help with something like /? or -h

[sselph]

There are a couple issues with the whole megadrive/genesis situation. First one is when I did the mapping from hash to gamedb id I didn’t really care which version I chose as long as there was a match. So if there were a US version and a EU version I just chose one at random, sometimes I looked to see which one had the best description or clearer image. The other issue is data quality from thegamedb, there are several megadrive games that have genesis art and possibly vice versa.

When I have time to remap MD and GEN I’ll take better care at only giving a MD version a GEN match if there isn’t a MD entry in the DB and vice versa. Ideally we could get the entries in thegamesdb fixed and improved so that other projects benefit as well.

I have tinkered with the idea of setting up a repository of my own to improve some of the MAME stuff but haven’t had time. If I do, I’ll see if I could do something similar for other systems but I imagine the cost would be prohibitive and I won’t actually do any of it :)

[sselph]

I don’t know enough about the pbp file format to know if I could translate the information it contains to what would have been in the original bin file to match it against the hash in redump.

[sselph]

They are installed just hidden from the menu if there are no roms in the emulator’s folder. If you add some roms in the folder for the emulator and restart Emulation Station the emulator should show up.

[sselph]

This scraper was designed to run inside or outside a system running emulation station so you have to provide the paths where the roms are, the images should be saved and where those will be on the target system so the gamelist is correct. I prefer having the gamelist and images all contained in the romdir because it makes copying and backing things up easy. The author of emulation station prefers having the 2 separated.

To run it on retropie and use the other paths you’d have to do something like:
PLATFORM=nes
scraper -rom_dir /home/pi/RetroPie/roms/$PLATFORM -image_dir /home/pi/.emulationstation/downloaded_images/$PLATFORM -image_path /home/pi/.emulationstation/downloaded_images/$PLATFORM -output_file /home/pi/.emulationstation/gamelists/$PLATFORM/gamelist.xml

[sselph]

My scraper, in the default mode, generates a completely new gamelist.xml file and uses the title given in the no-intro or redump database. If the entry was manually added, and not from one of those sources, it uses the title in thegamesdb(similar to the internal scraper). There are several flags to control the behavior.

[sselph]

Glad I could help. I’m in the process of creating a tool to make it a little easier to manually edit the gamelist files since they have this issue of being picky about allowed characters and encoding, etc. But just taking much longer than expected.

[sselph]

NP, that makes more sense. I updated my original post to avoid confusion.

[sselph]

Hi xenon-777,

I read the code and will attempt an English translation:

Usage:
scrapertool.py <command> [optional parameters]

Commands:
clean
Remove all scraped data. Accepts an optional parameter, which is the system to clean. ie nes, psx, etc.

rename
Renames all files to the name from the scraped data in gamelist.xml. Then removes the scraped data. This should make searching in the future more reliable.

all
Remove the platform information from es_systems.cfg. This will tell the scraper to search for the game title on all platforms.

org
Rollback the changes made by the ‘all’ command.

disk
For PSX add disk information from the file name in to the gamelist.xml This allows you to distinguish between the different disks for the same game.

[Author]

Posts