I wouldn't worry at all, your LO is small at the moment, so the movements may not be felt at all. If you are getting a nervous/butterfly feeling in your tummy, this is bubs moving too.
I know I didn't feel bubs move until I was around 18 weeks pregnant.
He should be fine, maybe just going through a growth spurt and is tired or if you have been moving about alot, he can't move or if you've been tense yourself, he won;t move.
Just relax, sit back and don't worry hunny, you're a lucky lass to have been feeling him for the past 4 weeks.
However, if you are really worried, give your doc/midwife a quick call xx