The story of the growth and expansion of the colonies and the early U.S. has been detailed in the previous sections. In summation, however, the story is surprisingly simple in its outline. During James Madison's life the population grew at an amazing rate, primarily from reproduction and only secondarily from immigration. When James Madison was born in 1851, the population of the colonies was about 1.2 million; when he died in 1836 the population of the United States was more than 15 million. In many ways, however, the population changed very little. The population remained about 80% free and White, and the White population remained overwhelmingly British in descent. The population remained young and evenly balanced between men and women. People lived in large household in rural settings throughout Madison's lifetime.

All these young people had to go somewhere to find more and better land and they did — west — with disastrous consequences for the Native Americans in their way. When Madison was born, the colonists occupied a narrow strip between the Atlantic and the Appalachians. He died in 1836, the very same year Davey Crockett and Jim Bowie died at the Alamo.

The settlers moved west on trails and rivers, and behind them political and economic elites, motivated by patriotism and profit, built roads, river improvements, and canals. Jefferson and Madison, with their narrow interpretation of the Constitution, opposed federal spending on internal improvements. However, private spending, state spending and the creative federalism of Albert Gallatin partially and slowly filled the void. The settlers, did not wait, however; they moved west through the gaps and river valleys. They formed new states and let the infrastructure follow.

As the young nation spread west, the North-South division opened up. Slavery disappeared from the Northeast and never existed in the Midwest while it spread in the deep South. The North grew rapidly and showed early signs of urbanization and industrialization, while the South grew more slowly and remained agricultural. The North was becoming the land of growth, opportunity and innovation; the South the land of status quo. When James Madison died in 1836, the conditions were developing for the Civil War, the defeat of the South, and the end of slavery.