Well if you smoke it may well effect the strength of your muscles, so although you may have a smaller baby (as Cassi says, it's no way certain), you will actually be reducing your ability to push the baby out. Think about it, the uterus is one massive muscle, each contraction of which opens up the cervix to 10 cm's dilation, out of which the baby comes. So if you do anything which interferes in that process you are making it worse for yourself. Also if you smoke, it will effect your absorption of oxygen, so in the third trimester when you're already feeling breathless, you will be making yourself feel so much worse.That will last for three months - why set yourself for three months of much worse health, it's a lot longer than labour! Also you will need as much oxygen as possible during labour, and if you've impaired your ability to absorb it, again you could be making it much worse for yourself.
I understand what you're saying about being a small woman, the best person to talk to about that is your MW, as I'm sure she will be able to put your mind at rest. Also bear in mind that if you choose, you can have an epidural so you don't feel the pain during birth - plus there are all the other options such as gas and air, pethidine etc. All women are different - both of my cousins found giving birth really easy - to the point that one didn't make it to the hospital in time as she found labour so bearable (btw, she is 5 ft 2 inches high, and her hips are probably about 30 - 32 inches), and the other one only just made it! Also bear in mind that your body is designed to do this, and that women do it all the time, all over the world, every day. The weight/height of the mother does have a bearing on that of her baby, smaller women tend to give birth to smaller babies.