Then don't beat yourself up about it. If this does end up being permanent, then make new memories while holding onto the ones he forgot. I think he'll appreciate that a lot more than what you're doing now.