You see the thing is is that it would not be an issue between me and my husband. Because I tell my husband everything. If I was to have dinner with another man (though I am not sure why I would but lets just say it was a co-worker and we went out for lunch) when I got home from work that day I would tell him "oh me and John went to Restaurant Name for lunch today". He would then most likely make a joke about it and we laugh and go on with life. Then if he was to start hearing rumors he would tell me about them and then both of us would be annoyed at the person running their month. If it was a friend they would no longer be a friend, or the friendship would take a hit not our marriage. If it was a family member they would be called out about the issue and told off by us for spreading stories. Cause this is just the kind of marriage we have together. So I understand that not every marriage is like that, which is why I said if you feel bad about having lunch with someone other then your spouse, then don't do it. What works for one marriage may not work for another.
I'm not telling anyone what to do about such things. Only they know what is right for them and not to judge other people who don't share your views. Had I know this was a topic about married people having dinner together I would have never gotten into it. I thought it was about the appearance of evil. Now I am caught up in all this marriage and dinner debate.