[sselph]

If I had to guess, In line 4 you have several non-straight apostrophes ’ instead of ' you could replace them with ' to be sure that they are correct. Or you could run these commands on the command line and they will remove some of the windows encoded values from https://en.wikipedia.org/wiki/Windows-1252

$ perl -i~ -pe "s/\x91/'/" gamelist.xml
$ perl -i~ -pe "s/\x92/'/" gamelist.xml
$ perl -i~ -pe 's/\x93/"/' gamelist.xml
$ perl -i~ -pe 's/\x94/"/' gamelist.xml
$ perl -i~ -pe "s/\x95/*/" gamelist.xml
$ perl -i~ -pe "s/\x96/-/" gamelist.xml
$ perl -i~ -pe "s/\x97/-/" gamelist.xml

[sselph]

Hi,

Yes the -append flag should be what you are looking for, although the scraper will skip downloading any images that already exist so should be fast to catch back up either way.

I have too many flags :)

[sselph]

Neogeo is a arcade platform and requires a special flag. There is a check for mame but not fba/neogeo so you can work around this bug in PiAssist by doing it yourself with

$ cd /home/pi/RetroPie/roms/neogeo/
$ scraper -mame

And probably need to file a bug with Death259 to handle neogeo.

[sselph]

Make sure to quit emulation station before you try and edit or delete the file and see if that helps.

[sselph]

XML requires things be encoded a specific way. When you edited it in a text editor one thing is that it used “smart” quotes. Where the opening and closing quote are different(“”) vs (“”) those get encoded using a special windows encoding which isn’t valid in XML. Another is the dash (—) vs hyphen(-). You could try and turn that off and just use a straight single quote.

The id/source things are just some extra data I store in a way that doesn’t disrupt ES.

[sselph]

@muffy
Looks like that xml was edited in windows at some point which did a number on it. I manually went in and fixed all the issues I could find.

@nolageek
The script be default looks for an xml file in the directory where it is run. So in your case the rom folder where you ran the scraper should have an xml. This is the first place ES looks for the xml. I told muffy to cd to the gamelist folder since I knew it wouldn’t have any roms in it so the -append would just read the xml see there weren’t any roms and write the file back out to disk.

[sselph]

Odd those don’t look like xml created from my scraper and since they have last played and no information I suspect they were added by ES to track that information. I just realized my append flag is going to skip those so I’ll need to add something to try and tell if it was scraped or just played.

[sselph]

Try:
$ sudo raspi-config

Then the first option is expand filesystem which will expand the image to fill the SD card.

[sselph]

I had to fix a minor bug for parsing gamelists made by ES so that is uploading as v0.9.5-beta but my script has a -append flag that will only add things that are missing from the gamelist if the folder you run the scraper in doesn’t have anything in it it will just read the file then write the file.

example:

$ cd ~/.emulationstation/gamelists/nes/
$ scraper -append

If you haven’t seen my scraper before it is at:
https://github.com/sselph/scraper
On the release page are binaries for most systems, you could do this on windows or on the rpi itself. Simple rpi instructions are here:
https://github.com/sselph/scraper#install-from-my-binaries

[sselph]

Seems like maybe ES isn’t able to parse some unicode that it wrote in an older version. I ran the gamelist through my scraper to sanitize it a couple different ways. First was just parsing and rewriting(-clean) and the second had all unicode removed(-nounicode). If that works, I can tell you how to use it to clean up the other ones.

[sselph]

I have issues tracking adding new systems on github. It is a function of: are there available hashes, what are the file formats, are there entries in thegamesdb.net, how many games, how busy I am, etc.

Feel free to add issues for each system but I can’t make any promises until I look more closely.

[sselph]

Hi socialretrogamer,

To minimize false positives, On consoles I’m not actually using the name of the file only the extension so you could name them 1.nes, 2.nes and it should still work. The scraper is using the rom data itself. It hashes it and compare it to a hash to ID mapping database I generated by hand for each system it supports.

So there are several reasons it may not have scraped a rom:

• Different ROM dump and therefore different hash. The Zelda2 could be bad, hacked, overdumped, or a rev no in the no-intro hashes I used.
• No entry in thegamesdb, for SNES there are 3385 No-Intro roms and only 1055 games in the GDB. With clones I matched 2434.
• No entry in my DB, because I have to manually add the hash>ID, I don’t automatically have new entries.

[sselph]

Emulationstation hasn’t seen much development since Aloshi got a job in March so I don’t think there has been any change to this issue.

[sselph]

Thanks!

I think I see the error and have submitted a fix and releasing a new version. Hopefully I get all the issues before I hit 1.0.0 :)

[sselph]

Yeah we can create a backup DB. The metadata I could probably download another dat file parse it and shove it in the same data store I’m using for history then point to images in another site or see how taxing it would be to host them.

[sselph]

Hmm those errors are from the mame scraper trying to parse the result of getting the URL and getting a response it can’t parse. Since it happens with different roms and in bursts might be some throttling or issues with the website.

[sselph]

It uses mamedb.com. It strips off the file extension and pulls the url http://www.mamedb.com/game/wyvernf0

mamedb.com uses .147 and wyvernf0 is .154

[sselph]

Thanks for the report. I’ll release a fix soon if I don’t hear any other issues.

Regarding bin/cue: The scraper will still scrape the bin file if there isn’t a cue file. How it works is it looks for cue files, parses them then gets a list of associated bin files. Then hashes files cue/track1/track2/etc until it finds a match and uses that. So if there isn’t a cue it will just treat the .bin as a binary and hash that like normal.

[sselph]

SD card write speeds are much lower than read speeds for SD cards and several things can affect it like the chosen block size, etc. Also the Pi maxes out around 10MB/s so that doesn’t seem like it will be an issue.

You show the output of the dd command but not what the command was. Can you show that. Also partitioning and formatting isn’t needed when you are just dd’ing an image. The image contains the partition and formatting information.

Also showing the devices on the system with something like the following would be helpful:
sudo fdisk -l
or
sudo blockdev –report

[sselph]

If there is an image named the same and in the location that the scraper wants to place an image it won’t overwrite it. It will just add the file name to the XML file.

The default location is
./images/<rom name>-image.jpg and ./images/<rom name>-thumb.jpg

(Although -thumb isn’t used by ES so probably should disable it.)

There are several flags to adjust where and how images are handled that you can see with scraper -h but these 3 are relevant:
-image_suffix, -img_format, -no_thumb

-image_suffix controls the text appended to the end of the rom name and defaults to -image like ES was doing.
-img_format controls the file type to save images as and defaults to .jpg so if you have png you can switch it
-no_thumb tells the scraper to not bother with the thumbnail image since it isn’t used and you don’t want the scraper to download a thumbnail from the external source.

[sselph]

remove the -download_images=false. The script checks for a local file and skips it if it exists so that it doesn’t do work twice if someone re-scrapes.

[sselph]

something like this but it is VERY important you make sure /dev/sdZ is replaced with the path of your SD card.

sudo dd if=/path/to/retropie.img of=/dev/sdZ bs=4M

The process would be similar to the following except with the retropie img instead of raspbian. There is also a GUI version above the command line version:
http://elinux.org/RPi_Easy_SD_Card_Setup#Using_the_Linux_command_line

[sselph]

That is what I was thinking it might be useful for. It doesn’t do any of the LCD stuff so you’ll have to write that portion yourself but it could call that script to tell it when you start and stop playing a game and what game and system it was. It is also a Go library but figured it might be easier to do this so people could write Python, bash, etc and this could communicate with them.

The arguments that are passed to the script are currently fixed but I can change it if someone decides they want to use it. I can make it more flexible or dump a blob of json. If you use the -web flag and open the page while a game is running you can see what information I am pulling out. It includes everything in the gamelist.xml about the game as well as information about the system.

[sselph]

Hi,

Never heard back so don’t know if you got this working but I needed a similar ability so I created a library to monitor the running emulator and rom. I went ahead and create a standalone binary for this use case. If you need to try it you can see how to use it at:

Script to Monitor Running Emulator and ROM for other scripts

It is alpha state right now.

[sselph]

You can add a -add_not_found to include games my scraper can’t find. It will add the empty entry and I think it will include the image you already have.

[sselph]

I think this should do it.
scraper -download_images=false -no_thumb -image_suffix=""

[sselph]

Bear in mind I haven’t actually tried this but found this which might work:
https://github.com/tuomasjjrasanen/python-uinput

One of the examples is generating keyboard clicks

[sselph]

I just added some support for PSX in bin/cue format. If you don’t mind, try upgrading the scraper to see if it works now.

[sselph]

Hash not found just means I can’t find that particular rom in my hash DB. It is either from a system I haven’t added, isn’t in no-intro DB, or is slightly different from the version no-intro has.

If there were permission issues you’d see ERR: error processing resident evil 3.bin: open resident evil 3.bin: permission denied.

If I had to guess from the rom name this is PSX which I’m in the process of adding support for. The script tries .bin because that is the extension for several other systems but I’ll be adding bin/cue support for PSX.

[sselph]

I can’t think of anything the script would be doing that would cause things to freeze, but I haven’t updated to the new version yet. Does this only happen if you use my scraper or does it happen with the builtin scraper as well?

Using a gamelist.xml file does cause it to use more memory but the rpi2 has plenty so there might a leak somewhere. Have you checked the ES log, think it is at ~/.emulationstation/es_log.txt

[sselph]

A couple things can be going on:
If you are using .zip files the contents are extracted when you start a game and deleted when you stop(i think). If you cut power to pi that could cause there to be a rom.zip and a rom.bin.
ES removes anything in parens from the names when displayed so ‘rom (US).bin’ and ‘rom (EU).bin’ could show up as ‘rom’

[sselph]

I haven’t played around with v3 beta but you could try running something like the following to find a file if something moved.
sudo updatedb
locate filename

Also ls by default lists the current directory which starts out as your home directory /home/pi but you can do:
ls /
or
ls /opt
etc to list other directories. / is the root of the filesystem similar to C:/ on windows.

[sselph]

It is available here:
https://github.com/sselph/scraper/releases/latest

[sselph]

Thanks for the detailed info. I think I see the issue, I’ve submitted a fix and pushing a new release. I wasn’t closing the response when the game wasn’t found so after too many of those the error about too many open files happens.

[sselph]

Sorry, I’ve been traveling and not checking the forums. My scraper, by default, places the gamelist.xml and images folder in the directory that you ran the scraper which is the folder with the roms.

Where/How did you run the scraper and do you have the roms loading from the usb? or is the there some syncing from usb to sd? I’m trying to understand what is listed in the gamelist.xml and if that might differ from where the actual images are. You can check the xml to see what is listed there.

[Author]

Posts