Performance Issues with es-scraper

[doc-thirst]

First off, I wanted to say thank you to the folks who provided this great piece of software, very cool. Also to many of the posters here who asked and got answers to many of my questions already.

Now to my problem…

I have a fairly large (NES) rom set, somewhere in the 1500 range. A ton of these are goofy user created games and whatnot so I haven’t dismissed this is a rom related issue, but I really don’t think it is, but in the spirit of full disclosure I thought I should mention it.

I’m able to set up everything without a problem to the point where I run es-scraper. Up until this point, everything seems fine, performance isn’t an issue, all the emulators popup quickly. However after I’ve “Re-scraped the roms” with es-scraper the system drags to a halt. It hangs when switching between roms and sometimes takes several sections to switch between emulators. Game performance doesn’t appear to be affected, just navigation of emulationstation.

During the es-scraper process, I do get a ton of “cannot resize image” errors, pretty much on each file. I did verify that I have python-imaging installed and updated.

After it runs, many of my images are giant both in visual size and file size (some are as big as 6meg). I’d imagine this is the problem, emulationstation is choking on these giant images. Which leads me to a few questions…

1) Any suggestion on troublehsooting/fixing these resize image error?
2) Is there a chance this problem is the number of roms and not the images?
3) I can transfer these files via the network, can I just resize them manually?
4) What is an ideal size (both physically and file size) for these?
5) Is there anything I can provide to the community which might help you all troubleshoot?

[motorhead]

Same problem here, but I don’t have a ton of roms. The roms scrapped two or three weeks ago seems to have the image resized. But yesterday, before a binaries update, I re-scrapped some more roms and I found two issues:

– Manual scrapping is not working for me, it was like auto scrapping… strange.
– The new images added where huge.

Problem with an update?

[doc-thirst]

Humm, hopefully that’s the case, I’m not sure what options I have as a end-user if it’s not. Are there are permission issues I might take a look at?

[kilgorefry]

I’m also thinking (hoping) it’s just a bug that came with an update that someone more skilled than I am will be able to fix.

I noticed it for the first time last evening when I tried to do a re-scrape.

From terminal, I’m seeing this,

Downloading boxart..
Image resize error

[doc-thirst]

Yep, that’s what I get too. It appears to happen on every rom, but several do have art once it’s finished, granted it’s hit or miss on how big it is, but it’s art none the less.

[motorhead]

I changed them manually, it’s a good option if you don’t have a lot of new roms… waiting for a fix.

Enviado desde mi GT-I9300 mediante Tapatalk

[doc-thirst]

any suggestions on image size?

[kilgorefry]

I believe 350 pixels is the maximum width that the ES-Scraper documentation recommends.

[newsworthy]

I ran ES_Scraper myself from the setup script two nights ago, and let it go to work on the 2000 ROMs I have installed. Took ages, but it too downloaded a variety of sizes for each one. Some perfect, meeting the 350px size, but many that were huge and cause browsing to be slowed right down.

Contemplating running it manually on each rom directory one by one, forcing the 350px limit using one of the variables in the script. Is this a wise course of action? Or am I going to have to settle in for a long time of scraping again?

[doc-thirst]

It sounds like manually scrapping each or manually resizing each (out side es-scrapper – using photoshop or paint) is viable, but this is a maintained project so there should be a fix at some point sooner than later.

I plan to just sit tight and wait for the author to check out the problem (it might not hurt to reach out to him via git and let him know there is a issue – he may not be aware). I took this opportunity to clean up my roms and copy of a fresh list. I’m going to keep an eye on the git page and watch for an update. If I don’t see a new release in the next few weeks, I’ll go the manual route.

[masterehm]

Hi,

I have the same problem and I opened a post [1]. Only now I saw this post here. And yes, somethings wrong with the image resize function in the python script.

[1] https://www.petrockblock.com/forums/topic/es-scrapper-emulationstation-peculiar-presentation/

[masterehm]

Riddle solved. There was a change [1] on Nov 20, 2013:
Import PIL instead of just the Image class since it doesn’t resolve on all platforms.

-import os, imghdr, urllib, urllib2, sys, Image, argparse, zlib, unicodedata, re
+import os, imghdr, urllib, urllib2, sys, PIL, argparse, zlib, unicodedata, re

Change PIL to Image in scraper.py [2] and resize works again.

[1] https://github.com/elpendor/ES-scraper/commit/37c34f32ef522df2902316318ea876c1a3c11798
[2] /home/pi/RetroPie/supplementary/ES-scraper/scraper.py

[doc-thirst]

[quote=3402]Change PIL to Image in scraper.py [2] and resize works again.[/quote]

Was this a type-o? Shouldn’t it be the other way around?

EDIT: OK I miss read this solution. The author changed it to PIL from Image to accommodate differnt platforms, which is what is causing our error.

By changing it BACK to Image (as detailed above) the problem is solved. Sorry for my lack of reading comprehension.

[smartroad]

Woohoo I shall try this tomorrow :D Chers for identifying the issue and sharing the solution :D

[motorhead]

Hi! Still not working for me… I have same errors resizing images, scraping its crashing, and manual scraping seems to be broken too…

[masterehm]

You changed manually the line from PIL to Image in scraper.py? If you use the scraper.py with the option -v, you get an Image resize error?

[motorhead]

Sorry, it was my fault, it’s not working if I try to run it under Retropie-Setup, but it’s working fine using commands directly.

Thanks!!

[FreezX]

The solution posted by Masterehm solved it on mine too. Solved?

[scurvykid]

Thanks everyone who found the solution, worked a charm!!

[petrockblog]

[quote=3429]Sorry, it was my fault, it’s not working if I try to run it under Retropie-Setup, but it’s working fine using commands directly.
[/quote]

Which command are you using now on the command line? I would modify the RetroPie Script accordingly.

[motorhead]

Im using this one for manual scrapping:

sudo python scraper.py -w 275 -m

[petrockblog]

Thanks for that!

Did you also try the “-newpath” option of es-scraper? I cannot make it work for some reason right now.

[Anonymous]

Just wanted to start off by saying thanks so much for all the work you have already put into this. Just ordered some cases and controller adapters. Looking forward to putting the whole system together.

I have downloaded the lastest RetroPie scripts and notice this is still an issue. Are there plans to fix this in the script itself later?

Thanks again!

[Anonymous]

Following this forum has gotten me the closest to getting it to work so far! However, running ES-scraper manually took me about 3 hours for NES games A thru C, and it completely froze in the middle of the C list. I read the advice on the ES-scraper documentation website about renaming ROM files using goodtimes. I downloaded the goodtimes and goodgui and converted all my NES file names. I’m going to retry this with a small sample manually and with a large sample in auto-scraper. It seems that ultimately I will have to get the auto scraping to work when I’m doing thousands of roms. While ES-scraper isn’t necessary to build a retropie gaming system I want to make something that looks complete since I’m making it for my brother’s birthday. Cheers.

[Anonymous]

Also, does anyone know if the ES-scraper works better when you set the memory to 256 or should I leave it at default? For some reason the auto ES-scraper used from the setup GUI recommends it is set to only 16.

[RetroMarine]

Join the club…I delt with all the same frustrations as you, trust me. Truth is, the only legit way to do the scraping is manually, and even then, you will have to pay attention to the roms it auto names (which it will do even though it’s supposed to be manual). Then you have to go back through the gamelist.xml to delete the games it renamed wrong. I manually scraped all my nes, snes, genesis, and psx roms. It took FOREVER…but I think it was worth it, because you only have to do each emu once, then save the entire directory to include the gamelist.xml to your PC, and you can just reuse it over and over again.

As far as the es_scraper crashing in the middle of your C list…did it create a partial gamelist.xml? If it did, open it up and see what game it failed to write. You may have a “C” game with a variable the scraper can’t read. For example, it happened to me with the Sega Genesis game André Agassi’s Tennis. The scraper couldn’t read the é. Once I deleted the game, the scraper was able to finish. Check it out and let me know what you find.

[Anonymous]

Thanks for the feedback! After I posted this I figured out only about 75% auto scraped correctly, and sometimes it was the popular titles that didn’t make it. However, this is when they were all named incorrectly and filtered through goodtimes into new names. I’ve since found a rom download site that tends to have the roms already named correctly. In the end I will probably have to manually scrape and modify those unidentified roms a few times through until I have the base rom list easily identifiable. I might just manually scrape them in sections and weed out the crappy ones or rename them until the final product at which point I’ll of course have to do it manually from start to finish at this point.

Two questions I am wondering is whether the SD card write speed matters (ie 4 vs 10) or whether or not changing the the memory so that the pi has 256/512 vs 16/512.)

Another question: does anyone know what the file naming conventions are that are most easily identifiable by the ES-scraper? It seems like the most successful ones had a (U), a (!), and sometimes PRG. I’m guessing that (J) means japan and that (U) means US. Not sure what the exclamation point means. I started downloading the roms from http://www.romnation.net/srv/roms/ since they appear to be named well. I’ve been going for standard (U) without (!) if its available, ignoring the (hacks) (j) (Unl) and (0# B#) labels.

If I had the list that ES-scraper uses to identify roms I bet this could be a lot easier does anyone know how to access this?

Many thanks for all the input and help!

P.S. I started deleting Doom, duke3d, apple2, x86, and cavestory so that the gui loads only the emulators/roms the user has access to since I’m doing a usb controller only interface. This leaves the input config left, which someone could accidentally run and reset the game-exit. Does anyone know how to disable the input config screen? Thanks!

P.P.S. I discovered with the naming convention that (U) means US and (!) means verified good dump, so I’m going to test the difference between (!) roms and roms without the (!) label.

[somail]

amccabe: what romset are you using?

If you are using nointro I can can post my xml’s which have already been manually scraped. That should save you sometime.

[Anonymous]

right now I’m starting with GBC, NES, and supernintendo. I’m just downloading the roms from romnation that are marked with “[U]” and “[!]” What are the romsets and xml’s? Thanks!

[somail]

Sorry I have to correct myself. I forgot my romsets are hyperspin compliant which means they differ a little bit from nointro ( same roms files but different naming scheme). My xml’s probably will not be able to save you any time.

[boxxle]

[quote=4250]
P.S. I started deleting Doom, duke3d, apple2, x86, and cavestory so that the gui loads only the emulators/roms the user has access to since I’m doing a usb controller only interface. This leaves the input config left, which someone could accidentally run and reset the game-exit. Does anyone know how to disable the input config screen? Thanks!
[/quote]

To get rid of the Input Config screen, do the following from either the command line (close down EmulationStation by pressing F4) or by SSH shell:

– sudo nano /home/pi/.emulationstation/es_systems.cfg
– go to the very bottom of the file and add comment hashes like this:

DESCNAME=Input Configuration
NAME=esconfig
PATH=/home/pi/RetroPie/roms/esconfig
EXTENSION=.sh .SH
COMMAND=/home/pi/RetroPie/roms/esconfig/Start.sh

to this:

#DESCNAME=Input Configuration
#NAME=esconfig
#PATH=/home/pi/RetroPie/roms/esconfig
#EXTENSION=.sh .SH
#COMMAND=/home/pi/RetroPie/roms/esconfig/Start.sh

– hit ctrl+x
– type ‘y’ then hit enter
– done!

This should kick you back out to the command line. You can always re-enable the input config by removing the hashtags.

[Marlboro Gold Regular Cigarettes]

Excellent read, I just passed this onto a colleague who was doing a little research on that. And he actually bought me lunch because I found it for him smile So let me rephrase that.

[InsecureSpike]

right now I’m starting with GBC, NES, and supernintendo. I’m just downloading the roms from romnation that are marked with “[U]” and “[!]” What are the romsets and xml’s? Thanks!

mate, please DON’T mention rom sites, I’m sure this is in a post somewhere Florian posted a few weeks ago,

Sent from my iPhone using Tapatalk

[kingvoxel]

I was not able to get rid of the annoying Input Config Screen. I am running ES2.0.0-RC1. In my es_systems.cfg file I only saw meta data for various eemulators. I didnt see the following:
DESCNAME=Input Configuration
NAME=esconfig
PATH=/home/pi/RetroPie/roms/esconfig
EXTENSION=.sh .SH
COMMAND=/home/pi/RetroPie/roms/esconfig/Start.sh

Are you using the same version of Emulation Station as me?

[Anonymous]

Making the xml lists is by far the most time-consuming part. You have to keep an eye on the scraper and catch ones that don’t register properly. Sometimes change I,II,III,IV to 1,2,3, or 4. Once you have the xml lists, the roms, the image files, etc. remaking the system only takes like 3-5 hours for me. I’m not into playing games since I don’t have the time but I like the challenge of putting this thing together. Sorry didn’t realize the rule about website names. Kingvoxel, I used the emulation station that came with the retropie image that was the latest back around march. I’m not sure if this would have been changed but there should be a text file that you can just add comment lines to take the input out. I recommend this if you don’t want the hassle of having to reconfigure your controls whenever you accidentally load that screen. I also figured out that the images for boxart take up different portions of the screen when run on different resolutions (720, 1080, etc.) and that there is a quick way of modifying the images to conform in terms of height vs width ratios which helps with the aesthetics. One problem I haven’t solved is that sometimes my buffalo gamepads will send a random signal to the pie which will essentially be a random d-pad signal, and when you have a long list that you are scrolling through this can be annoying. I was also able to combine an atrix 4g lapdock, some imported adapters and wires with this retropie to make a portable battery operated gaming system. PS does anyone know how to delete old posts? I couldnt find them listed in any of the profile stuff. Thanks.

[Author]

Posts