New version of sselph/scraper v0.9.0-beta

[sselph]

You can use the -missing missing.csv that will write all files that weren’t scraped to a file with an explanation. Either the hash was not in no-intro, the game matched but was missing in thegamesdb when I did the last matching, or the game is in thegamesdb but no image was found. You can open that file and add a column with thegamesdb ID and I can add those to the master csv.

If you think data in the existing master csv could be helpful you can download it from:
https://storage.googleapis.com/stevenselph.appspot.com/hash.csv.gz

[proxycell]

Thanks a lot Steven!

[vretro]

Is it possible to scrape a folder and use custom images which are in the /images folder (as long as they are named the same as each item)? – Also being able to use this technique on an unsupported system?

[robertybob]

[quote=117734]Is it possible to scrape a folder and use custom images which are in the /images folder (as long as they are named the same as each item)? – Also being able to use this technique on an unsupported system?

[/quote]

https://github.com/sselph/scraper/wiki/Flags

Presumably this is what you’re looking for?

-download_images
If false, don’t download any images, instead see if the expected file is stored locally already. (default true)

[vretro]

Bingo! Thank you Robertybob. Making it work on unsupported systems, using “-download_images false” would be my feature request then I suppose.

[robertybob]

[quote=117745]Bingo! Thank you Robertybob. Making it work on unsupported systems, using “-download_images false” would be my feature request then I suppose.

[/quote]

I’ve not tried it myself, but maybe

-add_not_found -download_images false
could work? The first flag, it seems, will add an entry to the gamelist.xml with no metadata but perhaps with the image if it’s found (?)

*edit* I wonder whether this may be of use:

-use_gdb false

Use the hash.csv and theGamesDB metadata. (default true)

I’m just speculating here, but if the games are from an unsupported system then the hash.csv won’t have any entries for them, so maybe if this flag is set to false it will still add the games to the .xml in addition to the other flags above?

[sselph]

Yes -add_not_found -download_images=false and if the scraper doesn’t know about the extension involved you can do -extra_ext=".new" Depending on the name of your files you may need to do -image_suffix="" to tell it to name the images rom.jpg instead of rom-image.jpg

[vretro]

Thank you Sselph!

I will try those extensions again soon, there seemed to be server issues when I first tried. These additional extensions are really handy.

[mferraci]

Hi Sselph, thanks for your work, I really like your scraper.

[quote=114832]You can open that file and add a column with thegamesdb ID and I can add those to the master csv.[/quote]

I want to improve the hash list with some psone game of my collection. Do you use the .cue hash or the first .bin of each game ?

Mathieu.

[g0nz0uk]

Great work Sselph. One question are there higher resolution pictures available, I use the nbba theme so images are large on the right but in mame the quality is very low, I used the thumbs only switch, should I delete the images and try without the thumbs switch?

[robertybob]

[quote=118920]Great work Sselph. One question are there higher resolution pictures available, I use the nbba theme so images are large on the right but in mame the quality is very low, I used the thumbs only switch, should I delete the images and try without the thumbs switch?

[/quote]

Yes, thumbs is for downloading low-res images – this is incase the Pi suffers a slowdown due to having to display large images with themes. You may find each image takes a second or two to load.

[robertybob]

[quote=118528]Hi Sselph, thanks for your work, I really like your scraper.

sselph wrote:

You can open that file and add a column with thegamesdb ID and I can add those to the master csv.

I want to improve the hash list with some psone game of my collection. Do you use the .cue hash or the first .bin of each game ?

Mathieu.
[/quote]

I believe it’s both .cue and all .bin files

[sselph]

[quote=118920]Great work Sselph. One question are there higher resolution pictures available, I use the nbba theme so images are large on the right but in mame the quality is very low, I used the thumbs only switch, should I delete the images and try without the thumbs switch?[/quote]

This is because the images at mamedb are low resolution.

[quote=118528]I want to improve the hash list with some psone game of my collection. Do you use the .cue hash or the first .bin of each game ?[/quote]

The .cue and the main(largest) .bin should work. I used to do all the bins but there are some duplicates across games so the main bin file should work.

[g0nz0uk]

Hello,

I’ve scraped mame not used the thumb_only switch but the images are quite low quality, around 14kb per image, how do I get them into a better resolution?

Thanks

[pausuri]

Hi Sselph, is there any way to prevent adding the region to every single rom?

Cheers

[sselph]

The default is to use the no intro name which includes the region. You can do -use_nointro_name=false to use the name from thegamesdb which doesn’t include a region.

[zerojay]

g0nzOuk: The answer is on this very page.

[neighbourhoodnerd]

Hi SSelph!

I ran the scraper today via RetroPie-Setup and it did a great job. Much faster than the inbuilt scraper and produced far more results.

Two quick questions:
1. I want to run the scraper from the terminal instead of using the RetroPie-Setup GUI so I can add arguments. Running the command scraper doesn’t seem to work though.
2. All my multi-disk games fail to scan. Here’s an example of how they are named in my library: Final Fantasy VII [disk 1 of 3].ISO. Is there any easy way to scrape these? I don’t mind renaming them but naturally I need a way to know which disk is which.

[sselph]

Doesn’t seem to work isn’t very descriptive, can you provide more information?

The scraper works for multi-disc games but doesn’t work for iso files since redump hashes are for bin/cue.

[neighbourhoodnerd]

Sorry, that was a dumb way for me to explain it :P

I’ve seen online that the command scraper is the one to use, but if I run it on my Pi it says the command wasn’t recognised. Even though I definitely have it installed and have used it (but only via the retropie-setup app).

[sselph]

Ah retropie doesn’t place the binary in the system path so you can’t just issue scraper but would have to give the full path. Unfortunately, I can’t figure out from the code where it places the binary so you might be better off just downloading a copy by following my instructions here:
https://github.com/sselph/scraper#install-from-my-binaries

Just remember you have to find the latest release value here:
https://github.com/sselph/scraper/releases

[neighbourhoodnerd]

Thanks! I got it working :)

[enigma]

I need some help with an issue. I ran your scraper on my NES library of 1600-ish games and it worked perfectly. But I deleted the list to run it again without ‘use-nointro-name’ this time and now I can’t get it to run at all. It checks for new hash and then I get this error ‘got 413 Request Entity Too Large response’.

Is it something on my side? I’ve tried running without any arguments and running it on a small list of 5 games as well, but no change. I’ve also tried waiting a week or so to see if anything would change, but it hasn’t.

[sselph]

That is an odd error. You are getting the error from downloading the hash or later? Can you me a send me a PM with the output of the script? That error means that a request you sent to web server was too large. This usually only happens if you are uploading a file which my script doesn’t do.

[Author]

Posts