I think it is just too personal to seek advice from anyone else. Me and my wife have gone through this so many times. It is not something straight forward to say this is how you will become family friends with someone! Yes, we should love our neighbors. At the same time, God has a perfect plan for each and everyone. Part of that is, whom we interact with. There are people out there who need you and there are people whom you need to be with. Lord knows that perfectly. In our flesh we often try to resist the plans God has for us. Because He wants to take things by faith and not by sight. He does not show us this is the way for us, instead asks us to trust in Him that the way He is taking us is the perfect one. What I am really saying is, pray about it!
If it is God's will and plan that you spend more time with them, He will change your heart. If not, He will take care of it! As simple as that. We have seen both happen in our lives. We have seen people change, we have seen us changing, we have seen people removed from our lives. Trust in God and seek His will. Actually, that is the biggest hurdle. Us putting trust in God.
Let me share a small testimony. My wife used to have weekly ladies prayer meeting at hour house. We were in apartment then. It was just a few ladies. Everything went well for few weeks. Then things did not go too well. More time went in ladies talk and very less in prayer. They all had kids in school. They would come in morning, spend some time talking and little prayer, have lunch (wife cooks for all of them), go for shopping, come back, have coffee or something, talk for some more time and leave home. My wife had a baby just then and my daughter was really young. I was so upset with the way things were going. Because it was simply not fair on my wife. She is doing it for right reason - to have prayer meeting. I felt like people were exploiting her. It got worst when they left their kids at home and went for shopping! I was so furious and was about to step in to stop it completely. My wife is very wise spiritually! (way better than me!) She said we should only pray about it. She was doing it wholeheartedly. She did have trouble and sometimes complaint. But soon she realized if that is what Lord has put in her life, she will accept it wholeheartedly. When I finally came to terms with it, it came to an end. We never stopped it. But things changed for everyone and they all stopped. It is a sad thing that a prayer meeting got stopped. But it was good in many ways because it was not really a prayer meeting that was happening! I don;t know if Lord was just being patient with everyone, giving them a chance to not miss out on a blessing. But He always takes care of His children