I'm sorry brother but Jesus, hung around prostitutes, tax collectors and other "sinners" on a regular basis to minister to them. You couldn't use that line of reasoning with Him, so you shouldn't be able to use it with us who are His body either.1Th 5:22.. Abstain from all appearance of evil.
Even though a man might not be doing evil, but if it looks evil, then don't do it. Having a "women" of the night being seen with a Christian brother alone is having the appearance of evil.