First, if as you say "
he is truly the man of God" then submit yourself to your Pastor, ask him the same questions you asked us, listen attentively, and do not
react to his answers but prayerfully and humbly
reflect on his answers. It is really none of our business. You are gossiping and giving a bad report on your church and pastor.
Second, if you ever get the opportunity to preach again, then simply preach the word of God and then be a doer of the word. You cannot be the Holy Spirit for the other members.
Third, pray.
Matthew 9:38 Pray ye therefore the Lord of the harvest, that he will send forth labourers into his harvest.
Finally, follow.
Matthew 4:19 And he saith unto them, Follow me, and I will make you fishers of men.
Follow Jesus Christ and grow in Him!