You will be fine hun, we have a cat called Smartie and we got her when Rhys my 11 year old, was just 6mths old. Rhys toppeled over in his bouncer seat one day trying to catch the cat, and so we had to buy a sturdy one for him after that!
We just do as you have said, and make a point of not allowing him to sleep on or in anything baby related, and she soon got the message.
My daughter gets excma and the cat hair does agravate that, but we leave folded fluffy or fleecy blankets on the end of her bed, (all inviting!) and she sleeps on there instead! (means for less frequent washing of Brionys pillow too, (where the cat likes to sleep!)