[thadmiller]

I’m going to guess the “confusing” part is the image extension discrepancy – if so, this may be due to other scrapers – I put in the effort to make sure this scraper plays nicely with the built-in ES scraper, but if another scraper was run, it could have created conflicting entries. The fact that running it on a fresh install produces good images, seems to confirm that possibility. But if you do find that my scraper is causing causing an issue, I’d like to know the steps to reproduce, so I can fix it.

As for the order the scraper processes – the order of systems in the es_systems.cfg defines the order the platforms are processed. The order the ROMs are processed within each platform folder is arbitrary (I’m not sorting it) – so the order is actually defined by the OS.

[thadmiller]

Ah, yes, subfolders would cause an (easily fixable) issue. I didn’t even know ES would process them, but now that I know, it’s been fixed.

Do another
$ git pull
and the scraper should be okay with subfolders.

I’d still like to rectify the image discrepancy you ran into, but I’m not able to reproduce, and can’t think of any reason why it would do that. However, I wonder if paths, now being correct for your subfolders, might straighten things out a bit (but you’ll either want to add the -f parameter, or remove your old gamelist.xml files so you don’t have old stuff hanging around).

[thadmiller]

I’ll update CTRL-C to cancel out of all scraping.

So, looking at your verbose output, it looks like all your paths are correct(?), but zero files were found in each directory. That explains why nothing is scraped, but I don’t know why it would be finding zero files unless the paths are wrong.

It’s odd that the gamelist contains a different extension than the actual file – it’s the same string that saves the file (technically, a rename, but whatever) that is written to the XML file. I’m not able to reproduce this, could you let me know a platform and ROM that didn’t work for you?

[thadmiller]

you should be able to hit CTRL-C and it will exit cleanly (saving what has already been completed)

[thadmiller]

I added some logging to the verbose mode while the script parses the es_systems.cfg file. For each <system> it should list name, path, platform, and ext (all the required data) along with the number of files found within the path (not exactly the ROM count, since it looks for any file rather than matching the extension, but it should be good enough for debugging).

Just update the script with a
$ git pull

and run with the -v flag
$ python scraper.py -pisize -v -l

[thadmiller]

Also, I just added those new arguments a couple hours ago – if you haven’t already, you’ll want to do another git pull to get those updates:

$ cd ES-scraper
$ git pull

[thadmiller]

hah, true – looks like it didn’t find any ROMs at all.

In that case, maybe you want to try the manual method (ignoring es_systems.cfg entirely)
$ python scraper.py -pisize -v -l -name mame -platform arcade -rompath ~/RetroPie/roms/mame -ext ".zip .ZIP"
(those would be the default values on my retropie installation searching for mame, obviously change your name, platform, rompath, and ext as necessary)

In the meantime, I’ll add some more logging so we can see what platforms and paths it found (the verbose section, right now, just affects the scraping, but it looks like you’re having issues before it starts scraping at all).

[thadmiller]

I’ve updated the script with optional arguments to ignore es_systems.cfg if the -name and -platform are manually specified. Ex:
$ python scraper.py -pisize -l -name mame -platform arcade

It’s a slightly more manual operation, but if these arguments are included, es_systems.cfg (and any of its issues) will be completely ignored.

@brakanje, I don’t think you’re doing anything wrong. I’m only TheGamesDB.net as the source, so there is a likelihood some of your games are missing (about 10 of my 170 games didn’t exist in their library), but if you have massive amounts of missing scrapes, I suspect something else may be the issue. If you wouldn’t mind running again with the -v option, and capture the output, I’d like to take a look at it to figure out what’s going on.

thanks

[thadmiller]

I agree completely, any text within a comment should be fine, but unfortunately that’s not the case http://www.w3.org/TR/REC-xml/#sec-comments. If you ask me, it’s a “broken” definition. I suspect others feel the same way, and that’s probably why some other XML parsers ignore the — within comments.

And you’re not coming off needy or demanding – I wrote the script (or modified, anyway) because I was unhappy with the speed and poor matching of every other scraper I could find, and I want it to be helpful for others too.

[thadmiller]

Your last error is due to a <system> (in your es_systems.cfg) missing a <platform> element – this is needed to determine what platform to scrape from.

I have updated the script to ignore any systems missing information needed for scraping (this won’t fix the fact that information is missing, but should be able to skip over it, allowing any other correct systems to be scraped), you can easily get the updated script by:

$ cd ES-scraper
$ git pull

As for the double-hyphens within comments, that is actually invalid XML, and it’s python’s xtree.parse choking on the invalid characters (the XML parser used by the original author). I may look at trying to use a different (more forgiving on errors) XML parser or parsing it manually, but either of those options would require rewriting a decent chunk of the script.

For an alternative, quicker solution, I’m going to add arguments to allow a path, platform, and extensions to be specified – this will be slightly more manual, but will allow the script to run, no matter how old or broken the es_settings.cfg file is.

Thad

[thadmiller]

The error you encountered is due to invalid XML in the es_systems.cfg

Have you updated the RetroPie Setup script and Binaries?

$ sudo RetroPie-Setup/retropie_setup.sh

UPDATE RetroPie Setup script
--restart script--
UPDATE RetroPie Binaries

I believe that will fix your issue (and, in my experience, it also fixes many other issues with some of the emulators). Alternatively, you could try to fix the XML manually – just go to the location the script states: line 101, column 237 (but be warned, there will probably be a number of issues with the initial config).

Thad

[Author]

Posts