From scripture we know:
Joseph is betrothed to Mary.
Then he finds out she is pregnant. He keeps her out of the public eye for safety reasons.
I am sure Mary tells him that this is God's son, but he has to think about it, for he is not sure what to do about it.
Joseph then receives a visit by an angel in a dream confirming what Mary has told him.
He does take her to be his wife but does not "know" her (sex), until some point after the birth. I am not sure how the tradition of Mary not having other children could be believed since Matthew 1:
24 Then Joseph, being aroused from sleep, did as the angel of the Lord commanded him and took to him his wife, 25 and did not know her till she had brought forth her firstborn Son. And he called His name Jesus.
So Joseph does "know" her later and usually that results in children. We have this confirmation in Matthew 13:
54 When He had come to His own country, He taught them in their synagogue, so that they were astonished and said, “Where did this Man get this wisdom and these mighty works? 55 Is this not the carpenter’s son? Is not His mother called Mary? And His brothers James, Joses, Simon, and Judas? 56 And His sisters, are they not all with us? Where then did this Man get all these things?” 57 So they were offended at Him.
Here we see Mary being named and also His brothers, along with sisters unnamed. Joseph does not get named here though, which seems to me to be a sign that Joseph had died prior to Jesus' ministry. If we figure that Joseph was in at least his twenties when he married Mary, then Joseph would have been in his fifties during Jesus' ministry, since we know Jesus was around 30 years old. For a carpenter, it was likely that he would have died by that age. We know people didn't live that long on average in the working class.
We also see Joseph when Jesus is twelve and is amazing the teachers in the temple. Mary and Joseph had left without Jesus, probably thinking He was with others in the caravan of family and friends which they traveled with. This is believable knowing Mary and Joseph would have been keeping after the younger siblings of Jesus. Myself growing up in a large family, the older ones often are left to venture about while the little ones are being tended to.