"Don't throw the baby out with the bathwater" is an idiomatic expression for an avoidable error in which something good or of value is eliminated when trying to get rid of something unwanted. A slightly different explanation suggests this flexible catchphrase has to do with discarding the essential while retaining the superfluous because of excessive zeal.