Ok, I'm just gonna say this right now..."who cares?" I mean really, you love him for him. He loves you for you. It doesn't matter what his age is or what he looks like for his age.
Ignore the rude comments of others, and go on with your beautiful relationship.
He needs to feel good about himself no matter what his age is and not matter what he looks like. You've found yourself a good man it sounds like and he's lucky to have you too.
He seems very respectful, hard working, honest, and adores you. He could have purple hair for all you care.