The most fundamental unsolved problems in galaxy formation revolve around "feedback" from massive stars and black holes. I'll present new simulations which combine new numerical methods and physics in an attempt to realistically model the diverse physics of the interstellar medium, star formation, and feedback from stellar radiation pressure, supernovae, stellar winds, and photo-ionization. These mechanisms lead to 'self-regulated' galaxy and star formation, in which global correlations such as the Schmidt-Kennicutt law and the global inefficiency of star formation -- the stellar mass function -- emerge naturally. Within galaxies, feedback regulates the structure of the interstellar medium, and many observed properties of the ISM, star formation, and galaxies can be understood as a fundamental consequence of super-sonic turbulence in a rapidly cooling, self-gravitating medium. But feedback also produces galactic super-winds that can dramatically alter the cosmological evolution of galaxies, change the nature of dark matter cores and ‘cusps’, and re-structure the circum-galactic and inter-galactic medium. These winds depend non-linearly on multiple feedback mechanisms in a way that explains why they have been so difficult to model in previous "sub-grid" approaches. This resolves long-standing problems in understanding even apparently "simple" galaxy properties like the mass-metallicity relation. Finally, I'll discuss where stellar feedback fails, and additional feedback, perhaps from AGN, is really needed to explain observations.