im so sorry youve been through all this, 3 years is a long time and can make you see things differently to someone on the outside so I understand why you are confused.
I was with a man like this for 10 years.
Im going to tell you something he done to me because your guy sounds so so much like my ex, it might be something he is capable of.
When our 2nd daughter was due, actually I was almost 2 weeks overdue, my ex had a visit from this german girl he had been writing to. He insisted they were just good friends, but told me not to phone him or contact him in any way while she was there (we never lived together). I asked him what to do if I went into labour should I contact him then and he said no. I ended up on my knees crying and beggin him not to do this to me, that I needed him with me if I went into labour, and he said the reason I shouldnt contact him is because I might disturb them in bed together. I was dumbstruck and couldnt understand how he could do this. He ended up agreeing not to sleep with her incase I went into labour and needed to contact him, but I was never to stop him from doing what he wanted again as I had no right to try and own him.
The stupid thing about all this was that once my daughter was born, he insisted on her middle name being that of the german girl. I stupidly agreed.
That was just one of many many reasons why I should never have been with him and was the last straw, we split up for the final time when that baby was just 6 months old.
I used to say to people "but hes not all bad, we have some good times, when hes nice hes really nice, hes such a good father blah blah blah" and you know what I really believed it.
He was a selfish, manipulative control freak. A complete tosser. But he manipulated me so much that I believed that the good times made it all ok.
It didnt and it doesnt, NO-ONE desreves to be treated with such dis-respect.
I really hope you can see what a destructive relationship this is hun, it doesnt matter if there are good times and you think he would be a good father, none of it matters when he can then be so damn nasty and selfish.
You deserve to be happy, and you will move on in life and eventually meet someone who really makes you happy, like I have, someone who respects you and your child and wants to make you happy not control you!
Please dont let it go on for as long as I did