The parents of James Madison Carlton are Wesley P. Carlton and Laura (Lawry). Other children Geneavia, Willie, Fanny, and Timmy. The children possibly grew up in Morrisville township.1850 Census Wake Co.

Madison and Wife Elizabeth King were married in 1870.
A few years before her death, around 1900 they were living in House Creek Township, Wake Co.

Shortly after her death, Madison remarried Mary last name?
In 1910, they were living in House Creek Township.

In 1920, they were living in Leesville Township, probably at or around Leesville road.

I hope this will help.