I recall reading once upon a time that the original idea for this exemption was that it was for literal scholars - a few hundred priestly intellectual sorts that were professional serious full-time Torah-studiers. But the exemption didn't have any specific criteria listed for what that meant, so the ultra-orthodox all wound up saying "yeah, I study the Torah all day too, so I qualify."