Skip to main content
M
mariam3104
October 9, 2024
Question

AI Skill Email Classification not classifying correctly

  • October 9, 2024
  • 2 replies
  • 0 views

Hello, my team and I have used the new AI skill for an email classifier and we have several questions regarding the process.
For some context, we need the AI to classify emails into three groups, lets say A, B and C.

To train the AI we uploaded: for group A around 139 eml files, for group B around 103 eml files and around 70 for group C.
Once it was "trained", it said it had an accuracy of 100% and it  concluded there were no errors in the training process nor the actual classifying process, but as seen in the picture, it seems the model was  only trained with few of the files we uploaded for each group. 

We were a little surprised when we took a look at the metrics since 1- they were extremely successful and 2- the ai didn't use all the files we uploaded from each type.

We still went ahead and tested it with new emails, 1 from group A, 2 from group B and another 2 from group C. The results showed all emails but one were classified as group A, and the other one was correctly classified into group B. All emails from group C were incorrectly classified into group A.

Can anyone explain what's going on? Why the AI only took a few of the files we uploaded for training? And also why even though the metrics show its been properly trained, its still not classifying each email into the correct group?

2 replies

M
mariam3104Author
October 9, 2024

Hi, small update, we changed it so the amount of emails from each group was the same because we realized it might've been overtrained so it would favor group A. And the emails from group C are still being incorrectly classified as group A but we fixed it for group B. 

matthew.shutt
matthew.shutt
October 9, 2024

1) Training files vs. test files

When training begins, the model divides the sample files into two groups: training files and test files. What you are seeing is a count of the test files.

  • The model uses training files to learn about the documents or emails you expect to encounter in production. From the set of samples you provided, the model selects a few files and analyzes the characteristics of those files to learn about how to classify it or extract fields within it. For example, if the model identifies that most of your sample documents for the Invoice document type contain an Invoice No. field, it can use this information when testing.
  • The model uses test files to apply what it has learned about the files and determine if its predictions are correct. The test files are the remainder of samples you provided, not used for training. If you uploaded 10 sample invoices and 7 were used for training, then 3 are used for testing.

https://docs.appian.com/suite/help/24.3/evaluate-ai.html

2) Mis-classifications

I'm glad you added more files to address one of the issues you found. I would encourage you to continue to evaluate your data for overfitting and ensure you have a comprehensive machine learning data set.

docs.appian.com/.../evaluate-ai.html

Terms & ConditionsAccessibility statement