I have no time for a break
You're legally entitled to attend all your antenatal appointments without any kind of discussion with your employer and employers must legally provide somewhere for you to rest when you're pregnant.
Your employer sounds like a total idiot, but please don't let them get you so stressed out. Your health and your baby's health are worth so much more than he is.