Over the last couple of decades, observational studies have progressed from the anthropology of nearby galaxies to a direct study of the early Universe, uncovering billions of years of cosmic growth and challenging galaxy formation models. The wealth of data from deep extragalactic surveys have revealed a picture where galaxies follow a relatively tight relation between star formation rate and stellar mass. This observed star formation sequence encapsulates information about feedback in galaxy formation and the evolution of gas density and gas accretion rates over cosmic time. All the while, there exists a growing population of massive “red and dead” (quiescent) galaxies that are no longer actively forming stars, falling far below the observed star formation sequence. The physical mechanisms responsible for quenching star formation and the buildup of the quiescent population remain poorly understood. Moreover, we do not have a cohesive evolutionary theory that ties together the observed structures and stellar populations of star-forming and quiescent galaxies. With a state-of-the-art compilation of space and ground-based imaging and spectroscopy, I will present a self-consistent empirical study of the sizes, stellar populations, and star formation rates of a complete sample of galaxies spanning the last eleven billion years. This novel data set combines deep, high-resolution rest-frame optical imaging with accurate distance measurements of both quiescent and star forming galaxies, making possible the first detailed studies of the early development stages of massive galaxies. These observations enable us to understand the physical mechanisms driving the growth of massive galaxies over 85% of the history of the universe, while also reconciling existing tensions with theoretical galaxy formation models.