I am one of those people who do not have vivid memories of pains.
Suppose I stub my toe. While the toe is hurting, I know what the toe’s hurting feels like. After it stops hurting, for a while I still know what that felt like. But I know it less and less well as my memory fades, until eventually I know very little how it felt like. The whole process might take only a few minutes.
Thus, that mysterious “knowing what it’s like” involving qualia is something that comes with a parameter that varies as to how well you know it.
This should worry physicalists. Thin physicalists should worry because it doesn’t seem that the fading corresponds to any knowledge of the underlying physical reality. Thick physicalists who think that Mary just acquires a new recognitional concept when she sees red should worry, because it does not seem that there is any gradual loss of a concept. I continue to have the same “that experience” concept (the demonstrative “that” points to the same past experience, and does so in a first-personal way) and the recognitional abilities it enables (I can tell if another pain is like that one or not), even as my knowledge of what “that experience” is like fades.
It’s also not completely clear what a dualist should say about the fading of the knowledge. Normally, when knowledge fades, what happens is either that we lose details (as when I forget much of what I once learned in school about the Metis uprising), or we find the dispositional knowledge harder to make occurrent. But the fading is neither of these. Maybe what is happening is that our present knowledge becomes a less good representation of what it is the knowledge of.