I can't imagine having a little caffeine back when you were pregnant would have any long term impact, particularly post-birth.

Remember too, most of our Moms (if you're older than 35) drank and smoked when they were pregnant, and while I wouldn't condone that, there are certainly millions of people for whom there was no apparent impact.

If you're breastfeeding, I would certainly avoid any caffeine after 2pm, I would make sure your baby naps at least 2 times during the day, and I would make sure they go to sleep in the evening no later than 9pm.