Scraper: little details to solve…

[jordimayor]

First of all, my intention is getting better scrapping. It’s not criticism to the awesome and well-done job from sselph <congratulations, mate!!>

When call scrap a game from metadata into emulationstation, they save image in the folder called .emulationstation/downloaded_images. But when I scrape a system, I go to console and roms folder and then run scraper -thumb_only that generates a gamelist.xml file and the images folder. That is a problem, because when you pack a system for backup purposes it has imagefiles in two diferent folders.

It is possible a bad config, but I’ve recently installed retropie 3.2.1 from scratch… I’ve only installed the scraper 1.04 from console…

Thanks in advance, and please, any suggestion will be welcome, because I’m not sure to be wrong…

[sselph]

The current version of emulationstation looks for the gamelist.xml file first in the rom folder and if it doesn’t exist it looks in the ~/.emulationstation/gamelists/(system) folder and finally /etc/emulationstation/gamelists/(system). The images can be anywhere since the gamelist tells ES where to look.

I prefer the gamelist and images in the rom folder since it makes it easier to copy files to and from the retropie. I actually wrote the scraper to run on my desktop and then I copied the files over to make upgrading systems easier.

ES’s author, Aloshi, has the opposite opinion and prefers that the gamelist and images not clutter up the rom folder so the internal scraper places things in the ~/.emulationstation folder.

My script can be invoked with command line flags to tell it where to find the roms and where you will be placing the gamelist and images. This is how it works when my script is invoked from the retropie-setup script. But keep in mind the gamelist can only be in one of three places if you expect ES to find it.

Something like this(with (system) replaced by the system) but you need to double check the paths since I’m doing this from memory. The image path is specified twice since this script was originally designed to be run from a different system, this allowed the host and target system to have a different folder location.

scraper -output_file ~/.emulationstation/gamelists/(system)/gamelist.xml -img_dir ~/.emulationstation/downloaded_images/(system) -img_path ~/.emulationstation/downloaded_images/(system) -rom_dir ~/retropie/roms/(system)

[cyborguk]

Sselph,

I’m having a little issue getting your scraper working. I’ve ran through the install guide on github.com and my pi2 has downloaded release version v1.0.4.

I’ve scrapped the Megadrive and Snes rom folders and it created the image folders which have many thumbnail size pictures but the gamelist.xml that got created in the roms fokder is empty.

For mame the folder again it created the images folder, but this time no gamelist.xml file.

Is there a command I can run to generate the gamelist.xml properly?

Not sure why it’s gone wrong.

I’m using Retropie 3.2.1.

Thanks

[robertybob]

[quote=110013]Sselph,

I’m having a little issue getting your scraper working. I’ve ran through the install guide on github.com and my pi2 has downloaded release version v1.0.4.

I’ve scrapped the Megadrive and Snes rom folders and it created the image folders which have many thumbnail size pictures but the gamelist.xml that got created in the roms fokder is empty.

For mame the folder again it created the images folder, but this time no gamelist.xml file.

Is there a command I can run to generate the gamelist.xml properly?

Not sure why it’s gone wrong.

I’m using Retropie 3.2.1.

Thanks

[/quote]

This is happening for me now on any system – started doing this a little over a week ago.

[cyborguk]

Must be a bug on 3.2.1?

[sselph]

Hi cyborguk,

The gamelist.xml is placed in the current working directory. This is the same directory where it is looking for roms and placing images so if those are working it should be fine. You can try adding -output_file /home/pi/RetroPie/roms/(rom_dir)/gamelist.xml to make it explicit. Permissions don’t seem to be the issue since it is able to create an images folder and write to it and it should give an error in that case. The other option is the scraper didn’t find any hashes which doesn’t appear to be the case since it downloaded images. I haven’t changed anything in my script in over a month except adding and removing some hashes in my database.

[sselph]

I just saw another message where someone said the gamelist was placed in their home directory. Check to see if that is the case, if so that would be strange and I’ll have to spend some time diagnosing that.

[robertybob]

This was happening to me using the Windows version of the scraper, I’ll try scraping using my Pi tonight and see whether it makes a difference :)

*edit* appears to be only certain Sega CD games which it affects (strange!).

[cyborguk]

Let me check tonight.

Where do I add

-output_file /home/pi/RetroPie/roms/(rom_dir)/gamelist.xml

This seems to be where mine is but the file is pretty much empty.

[cyborguk]

Ran this:

pi@retropie ~/RetroPie/roms/megadrive $ scraper -thumb_only -output_file /home/pi/RetroPie/roms/megadrive/gamelist.xml

No luck, images folder is created but no gamelist.xml

[cyborguk]

Anyone managing to get it to work under 3.2.1? I’m at a loss now with it sadly.

[cyborguk]

Formatted another SD card and installed 3.2.1 on it and put a few games in the Megadrive folder and installed the Sselph scraper manually as the experimental one doesn’t work and this time it worked.

So my main SD card doesn’t work, what could be the issue?

[Soullous]

The issue seems to be that gamelist.xml is being overwritten with whatever the built-in scraper created every time EmulationStation is restarted. It doesn’t matter if gamelist.xml is saved in the rom directory or in the ~/.emulationstation/gamelist/(system) directory. They both get overwritten. I’m not exactly sure where the file is coming from that’s overwriting it, though…

[Soullous]

OK, so I figured out my issue. Not sure if it’s the same issue that others are having, but, I came across this:
https://github.com/RetroPie/RetroPie-Setup/issues/807

Emulationstation seems to load all gamelists at startup and writes them back at exit. So if emulationstation is running and you modify a gamelist any modification will be overwritten.

So, it appears that EmulationStation cannot be running while you run the scraper. If you’re running it from SSH while EmulationStation is up, or if you try to run it from the RetroPie setup script / Experimental through EmulationStation it will scrape everything, but your gamelist.xml will get overwritten. So you need to shut down EmulationStation and drop to the console, run scraper (or sudo ~/RetroPie-Setup/retropie_setup.sh) from the command-line, and then when you restart EmulationStation again, everything should load properly.

Ideally, the scraper should be able to handle this by telling EmulationStation to reload the gamelist somehow. I’m assuming it’s possible, since the built-in scraper can do it. In the meantime, I’d suggest adding this bit of info to the README. Excellent job on the scraper, though. Works much better than the built-in one. I’d love to see this become the default, or even perhaps just merging the searching algorithm in with the default one.

[herbfargus]

The end goal is to have it incorporated into emulationstation but that just depends on how much free time sselph has, in the meantime I think it would be wise for us to kill emulationstation in the script before the scraper is loaded from the retropie setup script as to not run into the same issues mentioned above.

[petrockblog]

if you exit ES with f4 I think it doesn’t save any gameslists. I might look at changing the exit so you have exit + save and exit without saving options or so

[zigurana]

if you exit ES with f4 I think it doesn’t save any gameslists. I might look at changing the exit so you have exit + save and exit without saving options or so

I think this is correct.
While we are used to think that the gamelist.xml files are the endpoint, for ES they are of course only the stored values of an internal construct (in working memory) that contains all metadata of a given system. So when a certain system is exited (you go back to the system overview), it will store the current version of that internal struct in our newly scraped xml file. It does this to update the values for “# times played” etc.
The build-in scraper alters the internal info, which is then written to file as regular.

I would be willing to try and see what needs to happen in order to tie sselph’s scraper into ES nicely.
Not sure if actually making it part of ES would be the way to go, though. That would make further development for the scraper more involved, and we would lose flexibility. Maybe it’s better to let the scraper write to a second set of xml files, which are then somehow imported into ES?

[sselph]

Hi zigurana,

Yeah this is most likely what is happening.

As for adding my code to ES, I have submitted https://github.com/Aloshi/EmulationStation/pull/355 to ES and it is in their unstable branch. This code is somewhat out of date but it is there. Now that Go can create C modules it might be possible to refactor my code or create a small wrapper to allow ES to import it as an optional library so that it is kept more up to date. If you are a programmer and want to look in to this feel free and I’ll be happy to answer any questions.

The other solution to this is that ES was moving to using sqlite for storage and I already have some unchecked in code to also use sqlite. When this happens ES would no longer be dumping everything to the xml on exit which would probably solve the issue.

[Author]

Posts