It may help if you can figure out why they are telling you this, with the understanding that you decide what you want or are able to do.
Sometimes it is useful to discuss the problem/potential tension with a mentor. This may be your lead maternity carer or Well Child provider (depending on the age of your infant).
It’s up to you to decide what’s best for you and your baby.