scraper for NES and SNES using ROM hashes

[sselph]

Hi,

I was reinstalling retropie and had to re-scrape my rom directory. For fun, I pulled the full game list for NES and SNES from thegamedb, manually matched everything, and mapped all the hashes to the ID in the DB. Then wrote a quick program to scrape using this data.

I figured I’d share it in case anyone might find it useful:
https://github.com/sselph/scraper

[deaftolight]

where is it

[sselph]

Odd pretty sure I added a link it’s on github.com/sselph/scraper

[Haris]

Sselph can you write similar script for the other emulators too?

[sselph]

It is possible to extend the script. I’d have to write code to extract the rom information if the format has headers and other information then use a list of hashes to manually build the mapping from hash to ID. If there is a platform you would like me to try I could take a look.

[Haris]

Sure man. I would like to see it in:

Fba,turbograf16,mame,genesis,megadrive

atari2600,gbc

Pretty much for all i suposse:)

[Haris]

I get an

-bash: $: command not found

when i try to run $ go get github.com/sselph/scraper

why is that?

[sselph]

I wrote the script in go so you’ll have to install that to compile it. Many distributions have packages so should be as easy as something like:
sudo apt-get install golang
You’ll probably need to set a GOPATH environment variable:
http://golang.org/doc/code.html#GOPATH

But to make things easy I compiled versions for most platforms at:
https://github.com/sselph/scraper/releases

I recommend running this on a machine other than the pi but I compiled one for it anyway.

(-thumb_only flag can speed it up for testing since it is not having to download/process large images.)

[deaftolight]

I just get a bunch of errors:

SIGILL: illegal instruction
PC=0x3b484

math.init·1()
        /usr/lib/go/src/pkg/math/pow10.go:34 +0x28
math.init()
        /usr/lib/go/src/pkg/math/unsafe.go:21 +0x70
fmt.init()
        /usr/lib/go/src/pkg/fmt/scan.go:1164 +0x70
github.com/sselph/scraper/rom.init()
        /home/pi/go/src/github.com/sselph/scraper/rom/rom.go:83 +0x7c
github.com/sselph/scraper/rom/snes.init()
        /home/pi/go/src/github.com/sselph/scraper/rom/snes/snes.go:48 +0x70
main.init()
        /home/pi/go/src/github.com/sselph/scraper/scraper.go:395 +0x74

goroutine 2 [runnable]:
created by runtime.main
        /golang-1.0.2/src/pkg/runtime/proc.c:221

trap    0x6
error   0x0
oldmask 0x0
r0      0x7332f8
r1      0x0
r2      0x0
r3      0x2
r4      0x0
r5      0x0
r6      0x0
r7      0x0
r8      0x0
r9      0x324fec
r10     0x10840000
fp      0x732eb6
ip      0xcafebabe
sp      0xb6ebcf24
lr      0x3ba18
pc      0x3b484
cpsr    0x20000010
fault   0x0

[sselph]

I got similar issues when I tried compiling from the raspberry pi. The build command needed ‘GOARM=5 go build <blah>’ but seems the version of go in raspbian is older(1.0.2 instead of 1.1+ I’m using 1.2) and couple things don’t seem to work.

I pushed a version that fixes one of the issues but the library that resizes images doesn’t work so you’ll have to use -thumb_only so resizing isn’t needed. Use go get -u github.com/sselph/scraper to update the package.

I also cross compiled a version for the raspberry pi using 1.2 here. Or if you like you can do that yourself from an ubuntu machine by installing golang-go-linux-arm and doing ‘GOARM=6 GOOS=linux GOARCH=arm go build github.com/sselph/scraper’
https://github.com/sselph/scraper/releases/download/v0.1-beta/scraper_rpi.zip

[grassmunk]

I just wanted to say thanks for this! You’ve saved me a tonne of time on scraping material!

[Floob]

Is there anyway it could deal with Atari Lynx games in the future?

[sselph]

I actually added that over the weekend try v0.5.8-beta and let me know if it is working correctly.

[Floob]

Great! I’ll give it a go. Thanks :)

[Author]

Posts