Over the summer I asked this very question of my friend R. See, one of the couples at church, their newborn passed away. I don't even think he made 2 weeks of age. So I was very disturbed by the idea that the baby never had a chance at this life and what would happen to him now that he was gone. And my knowledge of the bible is very shallow at this point, so I ask these things of folks I trust. Anyway, he flat out said yes, that baby is in heaven.
I don't have any bible verses handy that says that though.
I'm going to pick on you today because I'm in a fairly bad mood.........just kidding

.
The point that our knowledge of the Bible might be shallow is one thing.....what has our walk with the Lord taught us about Him? How shallow is our walk?
There comes a point in our walk with the Lord when we need to have grown enough to say 'this is right' or 'that is wrong.....my Jesus is not about that sort of thing'.
If the Bible had instructions for every little thing, we could never lift it up let alone live long enough to read it through.
The question: Do babies go to heaven when they die? If the question refers to the physically immature body of the baby plus the soul, that is one sort of question.
If the question is 'does the soul of the baby go to heaven when the body (probably faulty) dies?' That would be a different sort of question.
Digressing for the moment, but still on track, there is an interesting passage in the Old testament that I want to throw in, not as a 'red herring' but as a good illustration of drawing on personal knowledge. I first read the following in the Niv: Deut 22:28 ,29. In that version, it reads that the virgin is raped. I said to myself, 'surely not! Jesus could not be the author of this injustice!' I then looked at the Nkjv and the Kjv and found that the scenario is different. The clincher being '
and they be found', not 'and he be caught'. this now is talking about promiscuity, seduction, but not rape. I know that Jesus would not treat an innocent girl as the Niv does. So, back to the question.
The soul of a baby who dies will be saved and will enjoy eternal life. The body will, like any other body return to dust.
The soul of the baby will be equipped to worship the lord and enjoy eternal fellowship with Him. That soul will not be left for all eternity an immature underdeveloped person needing a 2AM feed and regular Nappie changes.
In the same way as a 99 year old legally deaf and bedridden person will not inherit a dysfunctional body (+ wheel Chair)

for all eternity.
Trust Jesus........babies are not roasted in the eternal BBQ. Consider: Lev 18:21. Lev 20:2,3,4,5. Jer 32:35. See how the Lord views the evil treatment of children......no way is He going to discard precious babies.
end of rant.