He's got no right to dish our ultimatums like that, especially when you are pregnant and vulnerable. He isn't in charge here - relationships are a two way thing and if he loves you he will have to listen to what is right for you at this time. Don't let him call the shots, talk it through and don't be bullied. If you are going to move in together, then it shouldn't be because you didn't have any choice. It's the wrong foot to start on and you will spend you life worrying about when he's going to change his mind and kick you out. If he thinks he's the one in control here, then it sets your path up for the future.
It doesn;t sound to me like he's thinking about your feelings, just his own. You don't need that level of selfishness in your life right now so set him straight.
There's quite a lot to consider in your situation. Are you family only moving because of the baby? Would they otherwise be happy where they are if you moved in with your boyfriend? Would they get the deposit back? If you turned round tomorrow and said you were moving in with him, how would this affect your relationship with your family?
If I were you I'd have a big talk with your family, because it wouldn't be fair to have them move house, only for you to move out in a few weeks because of your boyfriend, unless they don't mind moving anyway. Moving house is a huge and expensive committment on their part, you need to be fair and upfront with them.
Good luck and let us know what happened...