Whether it's right or not, it would be hard to take concerns from other people without giving them at least some thought, so I don't think worrying about what people are saying means it's not right.
Are they concerned just about the age difference or are there other issues?
Can you both talk about anything? Do you both have approximately equal 'control' in the relationship (That's not the right word but I'm not sure how to say it, I mean are you both able to get your needs met? One of you not giving in to the other all the time)
I would think the main things to work out are whether you are heading in the same direction, spiritually, what you want for your lives, what you like doing for fun etc. Have you talked about all the decisions eg children, where to live, friends, seeing family, past. Do you have enough in common?
I think these things are more important than age. Premarital counselling is a great idea.