Yeah stopping the formula top up is best, you my find that you will be feeding alot more until you supply re-adjusts, also don't worry if his weight does drop slightly as I'm sure you know Breastmilk is less fatty then Formula.
Is your HV worried about his weight? mine always liked to keep an eye on lily (but she did have reflux which caused weight problems) just to make sure she was following her line, but she was great and saw lily alot and knew she was healthy in every way, she was just small.
Just keep reminding yourself and your HV breastfed babies normally are smaller and don't gain weight the same as formula feed babies.
Try to stand firm and as I said trust yourself as I know how some HV's can get, they get overly worried esp if they've not breastfed or delat with many who have Breastfeed and often end up getting the mother to change to formula , when they don't really want to and it's not really needed.
If you have any problems and need some support feel free to PM me or post on the forum and I will always do my best to help.
xSuzx