Harvesting

How much do you invest in harvesting in the early waves? I usually go for harvesting for the first two level ups (unless I get useful purple upgrades) and fertilizer if I don't use melee. Is there an optimal point? Should I get more for an easier late game?