Broadly speaking, I think most people see him for what he is - not the worst player to wear the white shirt, but nowhere near one of the greats either. An average, beige, inoffensive player, in an average, beige, inoffensive side.
Where the trouble starts is when you've got Grimes fanboys claiming he's the second coming of Leon, a Championship Team Of The Year contender, and then they roll out meaningless pass accuracy and "progressive pass" success rates to back up their "point". That's what gets people's backs up.