It must be an eye of the beholder thing. I am told the swastika was originally a symbol of good luck but I still won't wear one. I do however respect your view about this.The symbol came from a lost religion, but I believe that it is always the usage of a thing that gives it meaning. So modern pagans do indeed use it, though usually for various "good purposes", at least in their context. I think that God wanted us to steer clear of symbologies, because people tend to get really stuck on them. Giving the symbols more meaning than they deserver. Worshiping idols so to speak.
Again the eye of the beholder, when I see a cross I see an instrument of His death and mine. That does indeed give me the hope of new life.
So I guess if there is no definite way to look at most symbols then one must follow his conscience as long as it does not cause someone else to stumble.