Interesting question, wether facial aphasia has to do with memory.
I have an acute visual memory and I'm really bad at recognising people.
I remember a story of a child who didn't understand the news story about bank robbers covering their faces so they would not be recognised. The child didn't realise that to most people, the face is more important than the rest of the person when it comes to identifying who is who.
Or the joke about the guy who dealt with being naked in public by using a small wash cloth to cover his face as he walked naked through a crowd.
I think other people focus their ability to recognise and recall visual detail to the face alone, and I take in every detail of the whole person the face isn't more important. And looking at the whole person is not an effective way of recognising them because so much of that changes.