I would worry about it, your the same sort of size as me and I lost weight right through the first trimester, although I counld see my stomach getting slightly larger. I wasnt throwing up but my eating habbits changed (small amounts more frequently) and I went off chicken entirly.
The baby weighs very little in the first trimester, its only when you get to the second that you gain on average a pound a month. Be happy about it, Im feeling like a beached whale now! when people ask when its due, I want to lie and say earlier so they dont think 'god your huge'