I wouldn't worry too much, I do think there is too much propaganda these days making us pregnant women feel like we shouldn't be doing anything!
I too moved house in the early stages, before I even knew I was pregnant, and that never had an effect. The other week I was trying to push my car out...