Grading using artificial intelligence

[Rick-Rarecards]

Originally Posted by Case12

Btw, huge database is not what AI is about. It will develop and decide on its own....thus, AI. Large confirmed samples are needed to start. The expert confirmation at beginning is what counts. I don't know what the probabilities are today. 2 years ago they were around 70%. Probably higher now. At minimum AI could be used as a filter to improve efficiency for this app.

Btw, this start-up had the SW and HW up and running. The developers were DOD contractors. They were in the expert confirmation process, which takes time and experts. Also, expert bias is a big concern because AI will grow on its own with that bias. Biggest concern we were concerned with.
.

Develop and decide what? What is the task that was being solved? 70% what? Look at GPT-3 how many parameters does it have and how training examples did it need? AI is not magic, you need to be more specific with the claims.

All major tech companies have their own AI platform.

Of course and my claim is they overstate their value and abilities.

[Case12]

Originally Posted by Rick-Rarecards

Develop and decide what? What is the task that was being solved? 70% what? Look at GPT-3 how many parameters does it have and how training examples did it need? AI is not magic, you need to be more specific with the claims.

Of course and my claim is they overstate their value and abilities.

I agree with everything you say. And AI is not vudu
magic.

In this app we were translating unknown deaf sign language movements to text and voice to animated avatar with text and sound (and visa versa). Some of this is already available without AI. Our goal was the unknown body motions part.

It isn't just talk though They were in the process of developing it. Admitting only hardware prototype (but close enough for design patents and working sound/motion/display hardware) and software prototype (working to the point of needing expert image and motion feedback on known movement). These folk started from scratch and built to that point in two years. I don't know their progress since my leaving in March 2020.

I apologize for overstating. And you are absolutely correct that it depends on the app. The one they were doing was pretty tough though. (Unknown motion, image, sound, text, context).

[mrreality68]

Hi

I do not know enough about AI to know where the technology is at or not at. But I cannot believe we already do not have the technology with all the advances that are mentioned and the things AI already does things like

From Forbes Magazine

It can produce breathtaking original content: poetry, prose, images, music, human faces. It can diagnose some medical conditions more accurately than a human physician. Last year it produced a solution to the “protein folding problem,” a grand challenge in biology that has stumped researchers for half a century.

Not to mention that they already automating business processes, gaining insight through data analysis, and engaging with customers and employees.

So why would it not be able to do cards unless it is as someone mentioned all about the income stream of Cracking the cases to regrade?

[Peter_Spaeth]

Originally Posted by mrreality68

Hi

I do not know enough about AI to know where the technology is at or not at. But I cannot believe we already do not have the technology with all the advances that are mentioned and the things AI already does things like

From Forbes Magazine

It can produce breathtaking original content: poetry, prose, images, music, human faces. It can diagnose some medical conditions more accurately than a human physician. Last year it produced a solution to the “protein folding problem,” a grand challenge in biology that has stumped researchers for half a century.

Not to mention that they already automating business processes, gaining insight through data analysis, and engaging with customers and employees.

So why would it not be able to do cards unless it is as someone mentioned all about the income stream of Cracking the cases to regrade?

I think it's perhaps a false equivalency, sort of like asking if we can put a spaceship on Mars why can't we cure the common cold? It may be very well suited for the tasks you mention but that doesn't necessarily translate. PS no insult to anyone here but that it can diagnose better than a human doctor, based on my recent experiences, doesn't impress me.

[Snowman]

Part 1 of 3...

I apologize in advance for the ridiculously long string of posts that this is going to be, but I did warn you that this is a lengthy discussion, so I guess I'm at least partially delivering on that promise lol.

There is a lot of confusion here, and across the hobby, about what the term "AI" means and how it works. I'll try to clear up some of this confusion as this is my field of expertise. I write "AI" or "ML" code for work every day and have a strong understanding of how these algorithms work, why they work, where they are likely to go wrong and why.

First, I have to point out that while 'Artificial Intelligence' and 'Machine Learning' are perhaps kissing cousins, they're not quite the same thing. The problem of grading cards with computational algorithms is a 'Machine Learning' problem, not an 'Artificial Intelligence' problem. Artificial Intelligence is when you have recursive algorithms that make use of something like deep learning or ensemble models with deep neural networks where the algorithms can often learn from themselves. This would be something like Alpha Go or Alpha Zero (the best chess "player" in the world) or Deep Mind, or Tesla's self-driving cars that learn how to drive better through simulations of millions of miles. The people coding these algorithms set up the framework and outline the "rules" for it to be able to learn on its own and then sorta turns it loose on the problem. This is not applicable to the challenge of grading cards.

Grading cards is a 'Machine Learning' problem. Specifically, computer vision and classification. In order to understand why grading cards is not a problem well suited for machine learning, you have to first understand how these algorithms work and what their limitations are because there are some important limitations that I would argue render this technology borderline useless for the specific application of grading cards. I will go into this in more detail below, but for now, I'm just pointing out that I take issue with using the term 'AI' for grading. This is NOT an AI problem. It is a machine learning problem. However, "AI" sounds cooler, so "AI grading" it is, right? This is a marketing ploy. OK, I digress.


How computer vision works:

Imagine you have a photo of you and your family sitting down in the park having a picnic, surrounded by fields of green grass. Try to visualize the photo. You know how that image looks to you, but what does it "look like" to a computer? Everyone is familiar with the binary 1s and 0s at the operating system level of a computer, but let's pull back from that and try to interpret how a computer might see color and objects in a photo. Most of you are probably familiar with RGB colors, but if not, it's helpful to know that colors on your computer screen can be rendered using RGB color values, which range from 0 to 255 for Red, Green, and Blue. So every tiny little pixel in your family photo can actually be defined by it's RGB color values. Those pixels in your photo that are part of the green grass surrounding you all look something like [0, 255, 0] meaning 0 parts red, 255 parts green, and 0 parts blue. Now imagine how that entire photo could be represented in a giant map mathematically. Break it down into 3 different matrices or grids: one for red values, one for green, and one for blue. The matrix for red would have a ton of ~0s in it (pretty much everywhere that the green grass is located) but would have higher values in the center of the matrix which correspond to where the people are sitting since people have red color tones. The green matrix would have a ton of ~255 values in it since there is grass all around you, but it would have lower values in the center where the people are since people aren't green. Make sense?

OK, so now we have 3 different matrices, or think of them as maps if that helps, where each pixel from the photo has a corresponding color value. These matrices full of numeric values are what enable computers to "learn" from photos. What sorts of things can a computer learn from an image? Quite a lot actually. One of the primary ways a computer can tell that something is different about a particular section of a photograph is through something known as "edge detection". Edge detection makes use of some fancy math to identify where the edges of an object are located in the photo. So in our example here, one of the "edges" would be where the green grass meets up against the people in the center of the photo. The mathematical values are different on each side of this "edge", which helps the computer to detect that this is an important location in the photo, and it learns to pay attention to it. Make sense? Great. If not, well, I apologize for being a crappy teacher. But this is the gist of how computers see an image and how they use mathematics to identify key aspects of a photo (or a scan in the case of grading cards). If you're perceptive and you can visualize how these matrices of numbers might look to the ML algorithms, you can probably already see how this could be problematic for grading cards. I'll get into that below. It's a pretty lengthy discussion though.


How machine learning classification models work:

One of the most common machine learning problems that data scientists work on is building classification models which aim to classify (or "categorize", or "label") something as belonging to a particular class. A simple example of this might be to build a model to predict whether or not someone is Male or Female based on a set of attributes that the computer learns from. So we might train that algorithm by feeding it the height, weight, hair length, ring finger to middle finger length ratios, hip measurements, shoe size, eyelash length, how fast they can run, and how much time they spend each month shopping or talking on the phone (stereotypes be damned). Then we feed all of that data to the algorithm for each person and tell the computer whether that person is a male or a female. The computer would then learn what each profile looks like and would be able to provide probabilistic estimates of someone being a male or a female for any new data you threw at it. So a person who is 6'2", 195 lbs with a size 12 shoe that runs a 5.1-second 40-yard dash with medium length hair might get classified as having something like an 81% probability of being a male and a 19% probability of being a female according to the algorithm (note that these algorithms are almost never quite as confident as you might want them to be). In addition to binary classes like 'Male' and 'Female', or 'yes' and 'no', or 'true' and 'false' type problems, there are also what are known as multi-class classification problems. So this might be something like classifying whether an animal is a fish, bird, mammal, reptile, or amphibian. The output for a model like this might be something like 3% probability of being a fish, 12% bird, 7% mammal, 46% reptile, and 32% amphibian if you were feeding it with the data of a monitor lizard. Multi-class classification problems are much less performant than binary classification problems for obvious reasons. More options lead to more variance leads to more uncertainty, which equals more errors made by the computer when classifying.

[Snowman]

Part 2 of 3...


OK, great. Got it, how does that apply to grading cards?

The problem of grading cards can be broken down into several distinct classification problems. This could be classifying a card as 'authentic' vs 'counterfeit', or 'trimmed' vs 'not trimmed', or 'recolored' vs 'not recolored', or 'creased' vs 'not creased', etc. However, the problem of assigning a paricular grade is a multi-class classification problem. Theoretically, the computer would "learn" how to recognize what a 10 looks like, what a 9 looks like, an 8, a 7, ..., so on and so forth. Then it would assign probability estimates for each grade. In practice, this would probably look something like 1% 1, 1% 2, 2% 3, 5% 4, 7% 5, 13% 6, 18% 7, 30% 8, 16% 9, 7% 10, after which the card would be classified as an 8 since the highest likelihood associated with it was an 8 at 30%. However, you could think of it as only being 30% confident that it's an 8. That's not super helpful in practice, and it's precisely the type of output we would expect from a mutli-class classifier like this. And this is IF it's working well despite all the other numerous challenges which I'll outline below. As Rick-Rarecards pointed out, there are both technical challenges and practical challenges that prevent this from being a useful application for "AI" or machine learning. I believe these challenges render this problem borderline futile and at minimum a considerable waste of resources. I will outline these challenges as I see them below.

Technical challenges

Identifying a trimmed card:

One of the most important challenges to understand is what's required in order to build what we call a "training data set" that the computer can learn from. Let's take the problem of detecting a trimmed card as an example. In order for a computer to be able to classify a card as either 'trimmed' or 'not trimmed', it first needs to learn what trimmed cards look like as well as what non-trimmed cards look like. How does it accomplish this? It uses computer vision as I talked about above, leveraging convolutional neural networks and "edge detection" ("edges" in an image like where the grass meets the player in a photo as I described above, not the physical edges of a card) to detect anomolies in an image or scan. So you create a large database of images (thousands and thousands of images at minimum) where each image is labeled as being either trimmed or not trimmed. Where do these labels come from? Humans, of course (and likely the graders specifically). So they have to sit down and document millions of cards, one by one, marking it as trimmed or not trimmed, recolored or not recolored, scratched or not scratched, creased or not creased, etc. Pretty much anything you want to be able to detect about a card, they have to have a massive database to learn from and humans have to physically examine those cards and label the training datasets. Then you feed that dataset to the machine learning algorithms (or rather a highly skilled, and extremely expensive, team of data scientists does this) and after a bit of black magic, you then have a computer that is "capable" of identifying trimmed cards. I put the word 'capable' in quotes here because there's a major caveat that needs to be understood here, and this is perhaps the biggest issue of them all. The machine learning algorithms, without question, will not be as good at humans at detecting trimmed cards. In fact, they won't even be remotely close to as good as humans are. Here's the problem. First, remember that training data set that we created for the ML algorithms to learn from? Well, humans labeled it! So it's not going to learn what a trimmed card looks like, it's going to learn what a trimmed card THAT HUMANS ARE CAPABLE OF DETECTING IN THE FIRST PLACE looks like. Remember, there are countless trimmed cards that humans can't detect. If it passes through a human undetected and gets fed to the algorithm, the algorithm is told that this is what a non-trimmed card looks like, despite the fact that it was actually trimmed. But even worse than that, the algorithm is working off of a database of scanned images. Even if the scan is extremely high definition, it's still just an image from one single angle of the card, looking straight at it. Imagine if it were your job to detect trimmed cards, but you weren't allowed to hold them and feel the edges with your fingers, or even hold them and just rotate the card in hand at different angles, catching how the light bounces off the card with every rotation. All you had to work with was a scanned image of the card on your computer screen. If you think you'd be good at detecting trimmed cards just by looking at a scanned image of it, I promise you you're wrong. You might be great at detecting a botched trim job, but nobody can detect a good one. And if you think measuring a card always (or even often) tells you whether or not a card has been trimmed, again, you're wrong. It's plausible that an ML algorithm could be trained to detect a trimmed edge on a vintage card that has 3 frayed edges and one super straight, clean, smooth edge. But those aren't exactly difficult to detect to begin with, so this isn't much of a win.

Also, you can't just create one large training dataset of all cards. You have to have separate training data and separate ML models for all different types of cards, each training dataset requiring many thousands of cards at minimum, and likely hundreds of thousands of cards to be even remotely performant (good luck with that). All of these are labeled individually by hand. You can't train a model on 1950s Topps cards and then scan a 2018 Topps Chrome Shohei Ohtani and expect it to know if the Ohtani has been trimmed. It will say it's trimmed every single time because the Ohtani has sharp edges and the 1950s Topps don't. A grader knows to differentiate this, a computer doesn't. You would need to have separate datasets for each card type. And this is just the tip of the iceberg. Trying to teach an algorithm to detect trimmed cards is a fool's errand. And if you think all of these hurdles can be overcome simply by scanning every card with some sort of 360 degree spherical scans, LOL. Ya, good luck with that too. You'd then need a separate ML model for every single angle, and now your problem just exploded 1,000 fold.

Detecting a recolored card:

This one is tricky, although it is perhaps the most interesting project to work on of all the possible ML applications for detecting altered cards. But it's an insanely large problem to solve. Here, you'd likely need far more training data sets than you would even in the trimmed cards problem. There's more variation in card printing techniques, inks, surfaces, and especially images within the actual cards themselves than there is variation in card stocks or edges like in the trimmed cards example. Here, you'd almost have to have a separate training set at least for every single issue (so separate training data for 1952 Topps and another for 1933 Goudey, one for T206, etc.), and depending on performance, you may even need one for each individual card! And remember, every training set requires, at minimum, many thousands of cards scanned, but likely hundreds of thousands to be even remotely performant. So the more granular your requirements become, the less plausible this problem is to solve. But let's pretend for a moment that we could at least group together certain cards. Perhaps all 1952 through 1956 Topps cards could be used in one training set. Every time the ML algorithm sees a print defect, it's going to think that is suspicious and will flag that card as having a high likelihood of being recolored. And again, this suffers from the same problem as the trimmed training data in that it can only learn to detect cards that humans can already detect as being recolored to begin with. I suspect it would perform quite well at picking up the obvious recoloring jobs, but then again, do we really need help with those? Not likely. The hope would be that it would be able to identify cards that were recolored very subtly, ones that might slip through human grading, but if humans can't flag those to begin with, it won't be able to learn what they look like because the training data doesn't flag them as recolored. It says they're not recolored. But even if all the cards were correctly labeled in the training data, it still would have a very difficult time distinguishing between print defects, a piece of lint on the scanner bed, a damaged card, one that was in fact recolored, and even cards which just have abnormalities in the image itself. Especially with modern cards. This could be very problematic. Basically, any abnormality in the image could result in that card being flagged as having a high probability of being recolored. And if you tuned the algorithms to not be as sensitive to these abnormalities, then there would be a tradeoff that would result in more recolored cards not getting flagged. There are always tradeoffs in machine learning.

As a reference for how these algorithms work, and what level of performance one might hope for, there was a somewhat infamous competition on a popular data science website several years back that had machine learning experts all over the world competing to come up with the best algorithm to be able to detect whether a picture was of a cat or a dog. The winner was able to code an algorithm that was 97% accurate. On one hand, 97% sounds pretty impressive, but when you weigh its performance against a human, who would get it right ~100% of the time, then it's no longer all that impressive. These algorithms are great for automating away large problems where we just don't have the manpower to be able to go through every photo manually, one at a time. So if we had millions of photos to classify as cat or dog, and we didn't have the time or manpower needed to do it manually, and if a 3% error rate was acceptable, then it would be a huge win, potentially saving some company millions of dollars in costs. But for the problem of grading cards, you can't accept a 3% error rate on a problem as simple as this. And that's just for detecting if an image is a cat or a dog. If you're trying to detect something as challenging as trimming or recoloring, the error rates would be much, much higher. When it comes to grading cards, the need for accuracy far outweighs the need/benefits of automation.

Detecting edge and corner wear:

For a set like 1986 Fleer basketball that has deep red borders and white paper stock, an ML algorithm could probably learn how to identify what good corners look like and what soft or bad corners look like because the matrices would show clear edge detection differences where the red borders and soft white corners meet in the image. For this problem, edge detection "works". Same with 1971 Topps baseball and the black borders. It could easily detect white chipping along the edges of those cards as it would show up in the data of the matrix. However, take a card with white borders and white paper stock and scan the image and you can quickly see how the algorithms would fail to identify the chipping or bad corners because the scanned image does not have an "edge" to detect (white on white doesn't create an "edge" in an image that can be represented mathematically). For this reason, "AI grading" would certainly underperform the expectations/needs of any TPG.

Detecting surface issues:

If you've made it this far into my post, then I'm guessing you're already able to anticipate what the issues might be for this problem. First of all, many surface issues wouldn't even show up in a scan because a scan is only taken at one angle, and you often have to rotate a card at just the right angle to be able to see surface flaws. How many times have you bought a raw card on eBay that looked perfect even in a zoomed-in scan, only to have it show up with surface flaws or even wrinkles? It happens all the time because scans often don't pick up on these flaws, especially with modern chrome cards. Feed that image to an ML algorithm and it won't be able to see it either. But even if it could, it still would need to be able to differentiate between a surface flaw and just some random abnormality in the image itself. It would also need to be able to differentiate between a surface issue and the natural variation in paper stock for vintage cards. 1948 Leaf cards come to mind as those often have dark paper fibers that are visible even through the print. Also, think of the image in the card itself. Is that little speck a raindrop in the photo or a flaw? Is it from dirt on the photographer's lens? Is it a scratch in the surface? Is it lint on the scanner bed? A scratch on the scanner bed? You get the point. All of these things cause an increase in the error rates that an ML algorithm would produce. And again, it would underperform any human that's even remotely competent.

I should also point out that each of the use cases above are the ones that the TPGs are MOST hopeful about lol. In a recent interview, Nat Turner explained that they are not currently using Genamint technology specifically for grading cards, and that realistically, they probably only hope to be able to use it to identify altered cards or cards that have already been submitted for grading before. And even this lofty goal they aren't planning to achieve until the end of the year. "AI grading" isn't coming to PSA anytime soon, and I'll go ahead and go out on a limb here and say it likely never will. The problem of actually assigning a numerical grade to a card is considerably more challenging than any of these binary classification models above, and would produce considerably higher error rates than any of the issues above. It's just not an ideal application for machine learning. For some tasks, computers can be taught remarkably well how to do something. But it's not a magic solution for everything, and you really need to understand the problem you're trying to solve deeply AND have a deep knowledge of how these algorithms work to begin with in order to know if you have a problem that is well suited for machine learning.

This is what happens when executives get excited about technology that they don't understand and buzzwords like "AI" simply because everyone else is doing it, so why shouldn't they? It seems like everyone and their brother in the corporate world today thinks "AI" is coming to revolutionize their industry and that they just have to win the race and get there before their competitors do. But in reality, many of the problems they're trying to solve just aren't well-suited applications for machine learning. Hell, even Uber and Lyft both gave up on their automated driving projects, and that's a problem that is extremely well-suited for AI. There are no shortage of problems that AI and ML are going to solve in the near future, or industries that will be disrupted by these technologies. Grading cards just isn't one of them.

[Snowman]

Part 3 of 3...


Practical limitations:

Classification models such as these would come with extremely high error rates, and these types of ML algorithms are simply going to get it wrong far too often because of the natural variation that exists across different card types and card images and the limitations on the input data (high def scans only show so much). The confidence intervals that would accompany an ML model's predictions would be so wide that they would be borderline useless in practice. As Rick-Rarecards pointed out earlier, a model that outputs a 51% chance of your card being authentic just isn't very useful, and it's precisely the type of output that these types of models produce.

The data warehouse required just to support these types of projects is extremely expensive to build and even more expensive to maintain. Data scientists (or "AI/ML engineers", there are many names for these types of jobs) are also not cheap. Neither are the systems architects and database admins necessary for supporting them. Most skilled data scientists' salaries could probably cover 3 or 4 of the most experienced graders on a TPGs payroll, and they'll need an entire team of them to work on these problems. Also, the problems themselves just aren't all that interesting to work on from a problem-solving perspective, and good data scientists are often more motivated by being able to work on something novel and exciting than they are by just salary alone since any good data science job pays well and there's no shortage of interesting companies working on fun and interesting problems that are all competing for their talents. Keeping the good ones around won't be easy. Especially in California. Overall, the scale of the investment in such a project is difficult to exaggerate. The money PSA spent to acquire Genamint is just the tip of the iceberg. The juice will not be worth the squeeze. I'd wager everything I own on that.

TPGs already have enough boxes to check in their pipeline before a card gets returned to the customers. Receiving, research, grading, QC, and shipping. If they were to add taking high-definition scans and running numerous ML algorithms for each card to that pipeline, it could easily triple the time spend on every card. And for what benefit? Worse predictions than humans can already do? Maybe they hope to flag cards to potentially examine more closely? And what percentage of those would be false flags? A lot, that's for sure. Also, I guarantee it would just turn into a running joke with the graders. They'll just roll their eyes every time a math nerd comes up to them with a "questionable card" report. They'll quickly realize on day one that they are better at detecting this stuff than the algorithms are. It's just not practical. It's not the solution they were hoping for.

Also, who would be responsible for interpreting the models' outputs? Normally, this would be a data scientist who interprets model results for executives at other companies. Their expertise is needed to explain some of the anomalies, which there will be no shortage of. But to pay someone with that skill set just to interpret model results on every single card that comes through a TPG like PSA? Yikes. That's a pretty big ask. And if you had a non-skilled worker doing it, then you might as well just scrap the entire project.


What it could do well:

ML could be used to grade centering on all cards with a clearly defined border. It would be a fairly straightforward model to build and one that would be expected to perform well. Yay, I guess? How big of a win is this really though? Do you really need a machine learning model to tell you that a bordered card is off-centered?

However, non-bordered cards pose a much more challenging problem. You could train a model to learn centering by having it pay attention to how far from the edges the Topps logo is for a particular set, but then you're running into the problem again of having to build an entire dataset with tens of thousands of cards from just one particular set (something they rarely have) in order to create the training data it needs to learn how to identify what a well-centered card looks like from say 2022 Topps Chrome (or any other new set that it hasn't seen yet). This is just not practical. And you can't combine different sets with different logo locations into the same training data, because one logo might be well centered 3/16" from the top and left edges whereas a different logo for a different set might be well centered 1/2" from the top and left edges. Having non-uniform distances both being "well-centered" would confuse the algorithm.

Someone mentioned that ML could be used to identify which set a card was from, perhaps to help in the research stage of a TPGs pipeline. In theory, this is possible, but I'm not so sure this is an "ML" problem per se. You certainly wouldn't build a multiclass classification model with tens of thousands of different classes (here a 'class' would be a set of cards like 1987 Topps or 2019 Topps Chrome Sapphire Edition, etc.) because that's just way too many classes for a problem like this. I suppose you could try a different approach, but it seems like more of a matching algorithm type problem, not really a machine learning one that would require a training set of data to learn from.

Fingerprinting cards - Again, this isn't really "AI" or "ML". This is just a matching algorithm. Just like when the FBI "runs someone's prints" for a fingerprint match or dental records. You're basically comparing the numeric values in the RGB (or similar) matrix I mentioned earlier against other image files in a database and calculating something like the Euclidean distance between all vectors in the matrix to come up with a similarity score. When two images have extremely low, or near zero distance, they're probably a match. But again, this is just math and some basic coding skills, this is not machine learning and certainly not "AI" (although I suppose they are somewhat related fields).


Additional challenges that I didn't address:
- Autographs on cards
- Memorabilia cards
- Short printed cards that don't have enough copies to be able to create training datasets from
- Some cards are bowed, others are flat, this could distort the "edge detection" locations in scanned images
- Crossover submissions with cards currently in other slabs
- And a whole lot more...