I agree ... sometimes it's hard to correctly grasp what Jesus means. I have seen a number of people "cast out demons", only to have them return. I have come to understand that they did not cast the demons out at all, but it was a deception - which can happen to people who think they have the authority, but don't. Demons are only subject to the Lord Jesus Christ, and to the apostles of Christ to whom He gave that authority.
There is an example in Acts that is quite interesting ...
Acts 19
13 Then certain of the vagabond Jews, exorcists, took upon them to call over them which had evil spirits the name of the Lord Jesus, saying, We adjure you by Jesus whom Paul preacheth.
14 And there were seven sons of one Sceva, a Jew, and chief of the priests, which did so.
15 And the evil spirit answered and said, Jesus I know, and Paul I know; but who are ye?
16 And the man in whom the evil spirit was leaped on them, and overcame them, and prevailed against them, so that they fled out of that house naked and wounded.
17 And this was known to all the Jews and Greeks also dwelling at Ephesus; and fear fell on them all, and the name of the Lord Jesus was magnified.
18 And many that believed came, and confessed, and shewed their deeds.