Ashley,
I punched in Worth Nunley without a birthdate and the 1880 census shows him living at age 16 with his grandmother, Nancy Nunley in Grundy Co., TN. Her birthdate was shown as 1822, born in TN. The only Nancy Nunley I could find in prior censuses with a similar birthdate but born in TN was Nancy E or L Nunley born 1822-23 in Tn married to a Curtis R Nunley born between 1815-18 in Tn but living in District 11, Macon Co., TN The 1860 census shows a son, J C Nunley born 1841 in TN who was old enough to father Worth Nunley. Just a possibility of course but at least some hints for you to work on.