From the 1860 census I know that William was married to Kate Banks, and they had a 4mo old son Fisher Middleton, later Fisher Milton. I find no other records of them via census records. I'm assuming they were on a plantation at that time that could have changes from Middleton Plantation to GOD knows what name.
At some point, I assume after the end of slavery, they left Fort Adams. The name was changed to Milton and the story we've been told was was that my 2 times great grandfather walked away from the plantation, shortened his name to Milton and lived in Torras, LA. I verified this in the 1930 census, where they were known as Milton.
William Middleton has other siblings, possibly Sam Middleton/Milton. From William Middleton are the children, Fisher, Solomon, Mary, and Martha Milton. We are told Mary and Martha were twins. I located Mary's death notice which shows a dob of 3/1/1891. I can't locate her birth certificate. DOD was 2/1979. They lived in Torras, LA before migrating to New Orleans after the 1927 flood.