Perhaps it deals with the fact that he wanted to bid his family farewell, and didn't realize that he should first be preaching the Gospel, to them. he should have been willing to preach the Gospel to his family and then bid farewell, and than go and preach Jesus Saves, to others. For if all he was going to do was bid them farewell he was going to his house, without taking the message of Jesus to them, so he was already looking back at his pass, and forgetting that he was called to preach.
And Jesus said to him, No, man having put his hand to the plow, and looking back, is fit for the kingdom of God.
Except the Lord shall build the house, they labor in vain, that build it...
We find that God must build the house, or the labor is in vain, yet this young man, knowing that his house probably knew nothing of Jesus didn't even give thought that he should not bid farewell, but rather, first tell his house of the wonderful news of Jesus.
Maybe we could say that someone who doesn't first think of bringing the Gospel message to his/her family is worse than an infidel:
But if any provide not for his own, and especially for those of his own house, he hath denied the faith, and is worse than an infidel.
I Timothy 5:8
So I think Jesus is always for family, and after God, the family is of the upmost importance, but the man in wanting to bid farewell, was willing to let his family go to hell without knowing of Jesus; and as such was not fit for the kingdom.