Hunting rare games with artificial intelligence
Edit: Since people have recognized this can be used for all sorts of other things within the hobby (variant detection, special releases, etc.) I've made a Discord where we can collaborate in gathering training data for the NN.
Click here to join: https://discord.gg/sFAsG5j
Dear NFR collectors.
For many months now I've been pondering whether or not it is possible to distinguish a regular cartridge from a Not For Resale edition.
My hypothesis was that a lot of "regular people" (non-collectors), list their games for sale on eBay, without realizing they have a Not For Resale edition. I, myself has made (at current market price) roughly 5400 USD in the span of 4 weeks (if I were to resale these games), and the people that I have shared this with have also had great luck using this.
But until now there's been no way to find those rare gems without browsing through 15.000+ listings on eBay, Craigslist, Kajiji or whatever site you prefer.
Today we propose a solution for this problem. I assembled a training set of 300 different pictures of different NFR titles for Nintendo 64. Different games have different NFR boxes (e.g. Legend of Zelda: OT has a tiny square) while Perfect Dark has a rectangle. Therefore we needed a diverse dataset that included all variants of the different types.
Before going on with this thread, I'd like to acknowledge the work of aero for helping make this project possible, he's an expert in general adversarial networks, and coming up with unique ideas on data formatting. Without him this project would not be possible.
Perfect Dark NFR Label:
Legend of Zelda: Ocarina of Time
Before training a neural network model on the data, we had to augment the data, this essentially means transforming the image in various ways including: flipping it, rotating it, resizing it (typically to a lesser resolution), adding noise, etc. to make sure that the network is agnostic of the surrounding conditions, and properties of the photos (you never know what you'll encounter on eBay). Based on the length, shape and position of the label, we can also make educated guesses about what game it has found.
The dataset was on a server-grade (meant to be run in a GPU rack, so it has no cooling and sounds like a jetplane) Nvidia Tesla K80 graphics card that aero bought (I firmly believe he has an intimate relationship with the thing ). Enough with the explanations, let's get to the visuals. These were some of the early pictures I took during training at epoch 12/100 (meaning it wasn't even close to the fully trained network).
The way the network learns is by manually drawing a mask (polygon) around a cartridge to make it learn what a cartridge looks like, aswell as a the various Not For Research labels. After many hours of training it recognize these features even on cartridges it has never seen before, in any lighting conditions/angles/etc. The image below explains what is fed to the network. "Ground truth" means the mask that was manually drawn by a person, the output is the networks guess as to where the features are located (the network does not know what the image to the right looks like, it's purely rendering the mask based on feature recognition that it sees on the image to the left.)
I do not have images of the later stages of the training process, but as you can see it can detect non-standard NFR labels at weird angles. The fully trained network can perfectly mask out the cart + label.
Heres DK64 - a pretty rare cart.
Here's another image where it recognizes the NFR logo on a Super Mario 64 label. Keep in mind, it has to determine that the bedsheets (?) are noise, aswell as ignore the very similar looking Nintendo logo.
Here is an example of a regular 007 cartridge, that has no NFR label. As you can see, it correctly determines that this is in fact not a NFR cart.
This is how the network did at epoch 20/100.
If there's any interest in this I'll elaborate and perhaps me & aero can hook it up to a web interface and shre it with some avid demo game collectors.
The goal of this project was to get rare games out of the hands of people who may not take care of them as well as collectors do and ruin history. Oh, and also the financial aspect. It's essentially a nice little bonus ontop of my actual job.
This algorithm can be made to work with every "Not For Resale" game including DS, 3DS, Gameboy & SNES games aswell.
Hope this was an interesting read.
- nfr
Click here to join: https://discord.gg/sFAsG5j
Dear NFR collectors.
For many months now I've been pondering whether or not it is possible to distinguish a regular cartridge from a Not For Resale edition.
My hypothesis was that a lot of "regular people" (non-collectors), list their games for sale on eBay, without realizing they have a Not For Resale edition. I, myself has made (at current market price) roughly 5400 USD in the span of 4 weeks (if I were to resale these games), and the people that I have shared this with have also had great luck using this.
But until now there's been no way to find those rare gems without browsing through 15.000+ listings on eBay, Craigslist, Kajiji or whatever site you prefer.
Today we propose a solution for this problem. I assembled a training set of 300 different pictures of different NFR titles for Nintendo 64. Different games have different NFR boxes (e.g. Legend of Zelda: OT has a tiny square) while Perfect Dark has a rectangle. Therefore we needed a diverse dataset that included all variants of the different types.
Before going on with this thread, I'd like to acknowledge the work of aero for helping make this project possible, he's an expert in general adversarial networks, and coming up with unique ideas on data formatting. Without him this project would not be possible.
Perfect Dark NFR Label:
Legend of Zelda: Ocarina of Time
Before training a neural network model on the data, we had to augment the data, this essentially means transforming the image in various ways including: flipping it, rotating it, resizing it (typically to a lesser resolution), adding noise, etc. to make sure that the network is agnostic of the surrounding conditions, and properties of the photos (you never know what you'll encounter on eBay). Based on the length, shape and position of the label, we can also make educated guesses about what game it has found.
The dataset was on a server-grade (meant to be run in a GPU rack, so it has no cooling and sounds like a jetplane) Nvidia Tesla K80 graphics card that aero bought (I firmly believe he has an intimate relationship with the thing ). Enough with the explanations, let's get to the visuals. These were some of the early pictures I took during training at epoch 12/100 (meaning it wasn't even close to the fully trained network).
The way the network learns is by manually drawing a mask (polygon) around a cartridge to make it learn what a cartridge looks like, aswell as a the various Not For Research labels. After many hours of training it recognize these features even on cartridges it has never seen before, in any lighting conditions/angles/etc. The image below explains what is fed to the network. "Ground truth" means the mask that was manually drawn by a person, the output is the networks guess as to where the features are located (the network does not know what the image to the right looks like, it's purely rendering the mask based on feature recognition that it sees on the image to the left.)
I do not have images of the later stages of the training process, but as you can see it can detect non-standard NFR labels at weird angles. The fully trained network can perfectly mask out the cart + label.
Heres DK64 - a pretty rare cart.
Here's another image where it recognizes the NFR logo on a Super Mario 64 label. Keep in mind, it has to determine that the bedsheets (?) are noise, aswell as ignore the very similar looking Nintendo logo.
Here is an example of a regular 007 cartridge, that has no NFR label. As you can see, it correctly determines that this is in fact not a NFR cart.
This is how the network did at epoch 20/100.
If there's any interest in this I'll elaborate and perhaps me & aero can hook it up to a web interface and shre it with some avid demo game collectors.
The goal of this project was to get rare games out of the hands of people who may not take care of them as well as collectors do and ruin history. Oh, and also the financial aspect. It's essentially a nice little bonus ontop of my actual job.
This algorithm can be made to work with every "Not For Resale" game including DS, 3DS, Gameboy & SNES games aswell.
Hope this was an interesting read.
- nfr
Comments
Also, welcome to the forums!
Really cool read. I got to learn about how neural networks for a class I had to take last semester, really interesting stuff. It's neat to see the captured outputs. I guess my biggest question is how do you pull data from the listings themselves? Or is that something you have to do manually?
Also, welcome to the forums!
Hey there. Thanks for the positive comment.
Just some extra info - the training set is comprised of two parts. Here's what they contain:
The first dataset has roughly 300 images of Not For Resale carts taken from various angles, with various resolutions, these are the types of images you get when you fetch them off eBay, and that's exactly the type of data we're looking for. Real world photos as they would appear in an eBay listing.
The second dataset has a few more images ~650 consisting of regular NTSC carts (they're simply easier to find images of).
To verify that the network is able to distinguish between NFR and non-NFR carts we mix the datasets and measure the amount of correct guesses. I'm happy to say that we're currently sitting on a 100% success rate after the network was done training.
Now, to answer your question: I've written a crawler that has a list of all known NFR titles for the Nintendo 64. It fetches all images, from all listings, on eBay and alerts me if a listing pops up that the neural network with some certainty has determined to be an NFR cartridge. It also sorts by "newest listings" and alerts me via SMS if I'm not at the computer at the moment (I have this running on a Raspberry Pi with a GSM module + SIM card installed).
- nfr
It's fun to see something that feels almost futuristic (a la Altered Carbon or Black Mirror) applied to something as niche as finding NFR games, haha
Haha, yeah - but I couldn't think of a better way to solve this moderately hard problem.
Retro Nintendo games (amongst other items I know are valueable) is my fun little arbitrage Wall Street-type game. I have a piece of software monitoring the market and general price trends.
If there's interest I could share it with a select few collectors that have proven to be in it for the actual collection part and not the money - ironic, I realize. :-)
- nfr
Brilliant idea, that's awesome! I'm a little worried that you may have just started Skynet. It's only a matter of time before it becomes self aware.
Every morning I wake up, I fear I may accidentally invent Skynet.
I wish I could share more details about my personal life and the computer science work I do there, this was done over 2.5 days with 2 people working on it - but then again, I doubt any of you are excited about data as I am.
This technology can be expanded to any game, and using some formulas, you can automatically determine if an item has been mispriced. If you have an item you'd see tracked, give shoot me a PM and make me aware in this thread (I tend to miss the red alert telling me I have a message).
- nfr
Pretty interesting read. I know nothing about software programming and don't collect for NFR carts, but I can see some interesting elements that can be useful for other fields of collecting. Is the software ready for use yet?
Hello.
Yep. It can indeed - it can be used to spot misprints, rare editions, and any other feature that may be valueable, without the seller knowing.
If you have any suggestions, I'm currently in the process of making a web portal that is hooked up to this software, so a small sample of collectors can use it to get deals.
- nfr
Variant hunting would be nice.
Also, it would be nice to apply it to price chart applications to determine specific variants, specific condition, specific completion (CIB/Loose/missing pieces)
It would take the inhumane task of trying to document every possible variant of every game and automate it to the point of just having to verify things outside of the spec.
Really cool tech, I'd love to see this idea grow.
This is crazy! but not like randy quaid crazy, more like da Vinci crazy. So many other uses too. So cool
"So many other uses too", that's what people has been telling me. I decided to create a Discord. The admins of this forum is welcome to join and gain administrator privileges I have no desire to lead this thing, I just wanna write code.
If you want to join here's a link: https://discord.gg/Ncs33mQ
We can find variants all sorts of other rarities.
This is crazy! but not like randy quaid crazy, more like da Vinci crazy. So many other uses too. So cool
"So many other uses too", that's what people has been telling me. I decided to create a Discord. The admins of this forum is welcome to join and gain administrator privileges I have no desire to lead this thing, I just wanna write code.
If you want to join here's a link: https://discord.gg/Ncs33mQ...
We can find variants all sorts of other rarities.
Yeah when I’m on the computer later I’ll jump in just cause I’m curious. Right away my first thought was N64 box variants such as Mischief Makers, MM non-CE, Fighting Force, Toy Story 2, Gaunlet Legends.... list goes on and on and on
This technology is *REALLY* nice and I can think of a few applications for it beyond NFR hunting.
Variant hunting would be nice.
Also, it would be nice to apply it to price chart applications to determine specific variants, specific condition, specific completion (CIB/Loose/missing pieces)
It would take the inhumane task of trying to document every possible variant of every game and automate it to the point of just having to verify things outside of the spec.
Really cool tech, I'd love to see this idea grow.
This is a really good idea. However, it needs to be a collaborative effort, because in order to train something like this to recognize variants there's a few things I can think of that needs to be done:
- Gather images of every Nintendo 64 cartridge (or other console) in existance.
- Avoid images that has clear defects, such as sharpies, ripped labels, etc.
- Label each image NTSC or PAL
There's a ton of possibilities. Feel free to come up with suggestions.bad for collectors when the resellers with access to ebay APIs and a jump on buyitnow listings before everything else are able to insta-buy flippable items before anyone sees them hit the live server.
great for collectors trying to find unlisted variants
bad for collectors when the resellers with access to ebay APIs and a jump on buyitnow listings before everything else are able to insta-buy flippable items before anyone sees them hit the live server.
Oh don't worry. This technology is far out of their reach. If they want to comb through 15.000 listings for "nintendo 64 game" then be my guest. This tool was intended for collectors only.
- nfr
great for collectors trying to find unlisted variants
bad for collectors when the resellers with access to ebay APIs and a jump on buyitnow listings before everything else are able to insta-buy flippable items before anyone sees them hit the live server.
Oh don't worry. This technology is far out of their reach. If they want to comb through 15.000 listings for "nintendo 64 game" then be my guest. This tool was intended for collectors only.
- nfr
I feel that now that you've done it the cat is out of the bag, my guess is that there must be resellers capable of implemeting something like this, I would guess that a few already have some sort of IA to scout for deals, next step would be to implement something like this on top of that. Of course I don't really know what it takes to implement something like this in an efficient enough manner, but resellers that do this for a living will surely try to find a way to do it.
I feel that now that you've done it the cat is out of the bag, my guess is that there must be resellers capable of implemeting something like this, I would guess that a few already have some sort of IA to scout for deals, next step would be to implement something like this on top of that. Of course I don't really know what it takes to implement something like this in an efficient enough manner, but resellers that do this for a living will surely try to find a way to do it.
If they knew how to invent this tech, they would've done it long ago. Think about how much the Nintendo logo looks like the NFR logo. Now think about how similar these 2 look if we're dealing with a 600x600 image of a cartridge that's far away.
If they knew how to invent this tech, they would've done it long ago. Think about how much the Nintendo logo looks like the NFR logo. Now think about how similar these 2 look if we're dealing with a 600x600 image of a cartridge that's far away.
Looks like it's no easy task for sure, what are the chances of someone implementing a watered-down version of this? Use some pre-existing deep learning framework, have it look at full resolution images and looking only at new listings as they become live, is this somenthing that could be achieved? Of course if this can be achieved it won't be as good as yours but still it would be profitable.
If they knew how to invent this tech, they would've done it long ago. Think about how much the Nintendo logo looks like the NFR logo. Now think about how similar these 2 look if we're dealing with a 600x600 image of a cartridge that's far away.
Looks like it's no easy task for sure, what are the chances of someone implementing a watered-down version of this? Use some pre-existing deep learning framework, have it look at full resolution images and looking only at new listings as they become live, is this somenthing that could be achieved? Of course if this can be achieved it won't be as good as yours but still it would be profitable.
No I attempted a standard CNN that used the fullsize listing images, and it didn't detect (at least the way I modeled the data) the box accurately. It had a false positive for almost every image. Imagine scanning a bundled lot - that's gonna contain a lot of noise. Unless you have a large dataset, and a good way to structure the data, this isn't possible. You have however convinced me not to release the source code.
No I attempted a standard CNN that used the fullsize listing images, and it didn't detect (at least the way I modeled the data) the box accurately. It had a false positive for almost every image. Imagine scanning a bundled lot - that's gonna contain a lot of noise. Unless you have a large dataset, and a good way to structure the data, this isn't possible. You have however convinced me not to release the source code.
Sorry about that
Ok so it's not really easily achievable by the tools available at the moment, maybe later it would be, who knows. I love game changing (or game breaking) software like this, personally I've developed a couple of tools over the years to give me an advantage, nothing fancy at all, and certainly not AI.
Since I am not really familiar with the technology I'm just wondering about it, is looking for the NFR legend the simplest approach? How about instead of looking for the NFR legend, how about you first identify which game it is, once that is determined then compare with a NFR label of said game, maybe just compare the region where the NFR legend is expected, I wonder if that approach would be possible.
No I attempted a standard CNN that used the fullsize listing images, and it didn't detect (at least the way I modeled the data) the box accurately. It had a false positive for almost every image. Imagine scanning a bundled lot - that's gonna contain a lot of noise. Unless you have a large dataset, and a good way to structure the data, this isn't possible. You have however convinced me not to release the source code.
Sorry about that
Ok so it's not really easily achievable by the tools available at the moment, maybe later it would be, who knows. I love game changing (or game breaking) software like this, personally I've developed a couple of tools over the years to give me an advantage, nothing fancy at all, and certainly not AI.
Since I am not really familiar with the technology I'm just wondering about it, is looking for the NFR legend the simplest approach? How about instead of looking for the NFR legend, how about you first identify which game it is, once that is determined then compare with a NFR label of said game, maybe just compare the region where the NFR legend is expected, I wonder if that approach would be possible.
The thing is, there's only roughly ~25 (maybe a little more) N64 NFR carts, so there's no need to identify the game. You simply search for the title on eBay and they pop up.
The approach you suggested does not account for weird angles, torn labels, different ligting, etc. and like it would produce a ton of false-positives. This network is designed to look for features in the image, such as the shape of a cartridge (from multiple angles), aswell as the NFR label.
You could tilt the cartrige so the label is insanely stretched, but because it looks for features, and knows what a cartridge looks like, aswell as an NFR label from different angles, it'll have no trouble labeling it. This is a general solution to the problem of different camera quirks. However the suggestion you proposed has been tried before, and does (to some extent) work somtimes if implemented correctly.
- nfr
No I attempted a standard CNN that used the fullsize listing images, and it didn't detect (at least the way I modeled the data) the box accurately. It had a false positive for almost every image. Imagine scanning a bundled lot - that's gonna contain a lot of noise. Unless you have a large dataset, and a good way to structure the data, this isn't possible. You have however convinced me not to release the source code.
Sorry about that
Ok so it's not really easily achievable by the tools available at the moment, maybe later it would be, who knows. I love game changing (or game breaking) software like this, personally I've developed a couple of tools over the years to give me an advantage, nothing fancy at all, and certainly not AI.
Since I am not really familiar with the technology I'm just wondering about it, is looking for the NFR legend the simplest approach? How about instead of looking for the NFR legend, how about you first identify which game it is, once that is determined then compare with a NFR label of said game, maybe just compare the region where the NFR legend is expected, I wonder if that approach would be possible.
The thing is, there's only roughly ~25 (maybe a little more) N64 NFR carts, so there's no need to identify the game. You simply search for the title on eBay and they pop up.
The approach you suggested does not account for weird angles, torn labels, different ligting, etc. and like it would produce a ton of false-positives. This network is designed to look for features in the image, such as the shape of a cartridge (from multiple angles), aswell as the NFR label.
You could tilt the cartrige so the label is insanely stretched, but because it looks for features, and knows what a cartridge looks like, aswell as an NFR label from different angles, it'll have no trouble labeling it. This is a general solution to the problem of different camera quirks. However the suggestion you proposed has been tried before, and does (to some extent) work somtimes if implemented correctly.
- nfr
Thanks for answering my questions, I'm just floating ideas, most likely not going to try and develop anything but I find this very interesting
You mentioned the Nintendo logo causing a lot of false positives using a standard CNN, how difficult could it be to disregard bottom 20% of the label? Or to measure in percentage how far it is from the edges and determine that way if it is a NFR legend or the Nintendo logo. This most likely will not be valid for boxes, but I'd imagine more than 90% of the NFR listings found must be cart only and cart labels seem to have the Nintendo logo at predictable spots towards the bottom.
No I attempted a standard CNN that used the fullsize listing images, and it didn't detect (at least the way I modeled the data) the box accurately. It had a false positive for almost every image. Imagine scanning a bundled lot - that's gonna contain a lot of noise. Unless you have a large dataset, and a good way to structure the data, this isn't possible. You have however convinced me not to release the source code.
Sorry about that
Ok so it's not really easily achievable by the tools available at the moment, maybe later it would be, who knows. I love game changing (or game breaking) software like this, personally I've developed a couple of tools over the years to give me an advantage, nothing fancy at all, and certainly not AI.
Since I am not really familiar with the technology I'm just wondering about it, is looking for the NFR legend the simplest approach? How about instead of looking for the NFR legend, how about you first identify which game it is, once that is determined then compare with a NFR label of said game, maybe just compare the region where the NFR legend is expected, I wonder if that approach would be possible.
The thing is, there's only roughly ~25 (maybe a little more) N64 NFR carts, so there's no need to identify the game. You simply search for the title on eBay and they pop up.
The approach you suggested does not account for weird angles, torn labels, different ligting, etc. and like it would produce a ton of false-positives. This network is designed to look for features in the image, such as the shape of a cartridge (from multiple angles), aswell as the NFR label.
You could tilt the cartrige so the label is insanely stretched, but because it looks for features, and knows what a cartridge looks like, aswell as an NFR label from different angles, it'll have no trouble labeling it. This is a general solution to the problem of different camera quirks. However the suggestion you proposed has been tried before, and does (to some extent) work somtimes if implemented correctly.
- nfr
Thanks for answering my questions, I'm just floating ideas, most likely not going to try and develop anything but I find this very interesting
You mentioned the Nintendo logo causing a lot of false positives using a standard CNN, how difficult could it be to disregard bottom 20% of the label? Or to measure in percentage how far it is from the edges and determine that way if it is a NFR legend or the Nintendo logo. This most likely will not be valid for boxes, but I'd imagine more than 90% of the NFR listings found must be cart only and cart labels seem to have the Nintendo logo at predictable spots towards the bottom.
Hmm. I think your idea could maybe, potentially, get some results that are positive if you made a database of all possible NFR labels (Pokemon Snap, LoZ: OT, etc. has different NFR labels) and then moved that around in the x, y position and did a pixel similarity calculation. But even then, the lighting of the image vs. the label you're comparing to is gonna mess with your results. I think a better approach would be to look at the colors, since most NFR labels are a shade of red, see if you can make it detect rectangles that has sharp edges, with a red color. That's the only algorithmic way I can think of.
Also you're gonna need to transform images like this into a "square" image using linear algebra. If you use C# like me you may wanna look at AForgeNet (full disclaimer: I've never used it).
I would (and this is completely serious) like to see your progress using a purely algorithmic approach. It could be interesting to see if it could be done.
By the way, if you need help or have questions. Feel free to PM me or join the Discord I created for data gathering: https://discord.gg/Ncs33mQ
That is pretty awesome. Really surprised to see this as it feels like it came out of nowhere! I am pretty sure there are so many out there that never get discovered even rare ones. I only need a few left for each set from Nintendo and it is driving me nuts all these years not being able to complete them due to how rare some of them are. Good luck on this project and I am sure a lot of people would have a great use on this who are still far away from a near complete NFR collection.
Haha, glad you enjoyed the read. I'm new here, but I can see it's a bit different from the normal post (I hope it's allowed? Please delete the post @mods). The cool part about this tech is that it can not only identify NFR labels, it can also be trained to find fx. 3 vs. 5 screw games, cartridge variants, etc. It is very good at feature recognition with proper training data.
- nfr
This is an awesome idea! Im not smart enough to completely understand this but if i wanted to use this to find other variants of like snes games id just have to add a crap ton of pictures to a database of some sort?
Essentially yes you'd just feed it a "crap ton of pictures", mask out the features you're interested in (like the Not For Resale label), and let it train. If you formatted your data properly it'll with close to perfect accuracy be able to tell whether or not the picture it is fed contains the desired variant. :-)
- nfr