This was the only result I found in AL in the 1880 census for Benjamin Walton. Maybe a possibility? Lauderdale Co. is two counties to the west of Madison Co.

Source Citation: Year: 1880
Census Place: , Lauderdale, Alabama; Roll: 18
Family History Film: 1254018; Page: 153C; Enumeration District: 150.

Thomas Walton, 63
Mary Walton, 44
James Walton, 14
Martha Walton, 12
Mary Walton, 10
Sarah Walton, 8
Thomas Walton, 5
Benjamin F. Walton, 1

If you have names for any older siblings, the family can often be found in census records, even though your person wasn't born yet.