scraper for NES and SNES using ROM hashes

This topic has 13 replies, 4 voices, and was last updated 9 years, 10 months ago by Floob.

Viewing 14 posts - 1 through 14 (of 14 total)

Author

Posts
07/13/2014 at 03:47 #15067

sselph
Participant

Hi,

I was reinstalling retropie and had to re-scrape my rom directory. For fun, I pulled the full game list for NES and SNES from thegamedb, manually matched everything, and mapped all the hashes to the ID in the DB. Then wrote a quick program to scrape using this data.

I figured I’d share it in case anyone might find it useful:
https://github.com/sselph/scraper

07/13/2014 at 22:34 #15363

deaftolight
Participant

where is it

07/13/2014 at 23:15 #15372

sselph
Participant

Odd pretty sure I added a link it’s on github.com/sselph/scraper

07/13/2014 at 23:44 #15384

Haris
Guest

Sselph can you write similar script for the other emulators too?

07/14/2014 at 01:53 #15411

sselph
Participant

It is possible to extend the script. I’d have to write code to extract the rom information if the format has headers and other information then use a list of hashes to manually build the mapping from hash to ID. If there is a platform you would like me to try I could take a look.

07/14/2014 at 07:15 #15483

Haris
Guest

Sure man. I would like to see it in:

Fba,turbograf16,mame,genesis,megadrive

atari2600,gbc

Pretty much for all i suposse:)

07/15/2014 at 11:25 #15850

Haris
Guest

I get an

-bash: $: command not found

when i try to run $ go get github.com/sselph/scraper

why is that?

07/15/2014 at 18:40 #15976

sselph
Participant

I wrote the script in go so you’ll have to install that to compile it. Many distributions have packages so should be as easy as something like:
sudo apt-get install golang
You’ll probably need to set a GOPATH environment variable:
http://golang.org/doc/code.html#GOPATH

But to make things easy I compiled versions for most platforms at:
https://github.com/sselph/scraper/releases

I recommend running this on a machine other than the pi but I compiled one for it anyway.

(-thumb_only flag can speed it up for testing since it is not having to download/process large images.)

07/16/2014 at 01:04 #16067
deaftolight
Participant
I just get a bunch of errors:
```
SIGILL: illegal instruction
PC=0x3b484

math.init·1()
        /usr/lib/go/src/pkg/math/pow10.go:34 +0x28
math.init()
        /usr/lib/go/src/pkg/math/unsafe.go:21 +0x70
fmt.init()
        /usr/lib/go/src/pkg/fmt/scan.go:1164 +0x70
github.com/sselph/scraper/rom.init()
        /home/pi/go/src/github.com/sselph/scraper/rom/rom.go:83 +0x7c
github.com/sselph/scraper/rom/snes.init()
        /home/pi/go/src/github.com/sselph/scraper/rom/snes/snes.go:48 +0x70
main.init()
        /home/pi/go/src/github.com/sselph/scraper/scraper.go:395 +0x74

goroutine 2 [runnable]:
created by runtime.main
        /golang-1.0.2/src/pkg/runtime/proc.c:221

trap    0x6
error   0x0
oldmask 0x0
r0      0x7332f8
r1      0x0
r2      0x0
r3      0x2
r4      0x0
r5      0x0
r6      0x0
r7      0x0
r8      0x0
r9      0x324fec
r10     0x10840000
fp      0x732eb6
ip      0xcafebabe
sp      0xb6ebcf24
lr      0x3ba18
pc      0x3b484
cpsr    0x20000010
fault   0x0
```
07/16/2014 at 01:55 #16071

sselph
Participant

I got similar issues when I tried compiling from the raspberry pi. The build command needed ‘GOARM=5 go build <blah>’ but seems the version of go in raspbian is older(1.0.2 instead of 1.1+ I’m using 1.2) and couple things don’t seem to work.

I pushed a version that fixes one of the issues but the library that resizes images doesn’t work so you’ll have to use -thumb_only so resizing isn’t needed. Use go get -u github.com/sselph/scraper to update the package.

I also cross compiled a version for the raspberry pi using 1.2 here. Or if you like you can do that yourself from an ubuntu machine by installing golang-go-linux-arm and doing ‘GOARM=6 GOOS=linux GOARCH=arm go build github.com/sselph/scraper’
https://github.com/sselph/scraper/releases/download/v0.1-beta/scraper_rpi.zip

11/29/2014 at 10:52 #83195

grassmunk
Participant

I just wanted to say thanks for this! You’ve saved me a tonne of time on scraping material!

01/12/2015 at 22:54 #85057

Floob
Member

Is there anyway it could deal with Atari Lynx games in the future?

01/12/2015 at 23:10 #85062

sselph
Participant

I actually added that over the weekend try v0.5.8-beta and let me know if it is working correctly.

01/12/2015 at 23:30 #85064

Floob
Member

Great! I’ll give it a go. Thanks :)
Author

Posts

Viewing 14 posts - 1 through 14 (of 14 total)

The forum ‘Everything else related to the RetroPie Project’ is closed to new topics and replies.

Login