User Filter has too many options, I need to modify it to have only right amount from the Grid. How can I add inputs to the User Filter?

so as you can see in the grid below. There is only two items on the entire grid. 

But look at the options for the GRID Filter. There are over 50

The reason that there is more than 50 items is because is getting the entire set of Assignees on the entire Record. However this an specific item from the record, where there is only 2 items on this record, There is no need to display all those other options on the Record Type User Filter. Is there a way to do this? Or am I obligated to create a Costume Filter?