This is from the records of a relative that researches the Blantons and Williamson - part of a letter written in 1912 by one of James Williamson's grand-daughters to another of his granddaughters:

"Well cousin, you asked me how come Mr. Henry M. Goff is any relation to us? Well I don't know that he actually is, though he is said to be your father's (James Haynes Williamson) and my mother's (Celia Alice Williamson Blanton) half-brother. Ha! Ha! .... I have heard Mother say our Grandpa Williamson took him when he was a small child and raised him for as long as he (James Williamson) lived. There were 2 of the boys that were laid to Grandpa. The other one was named Azariah, and their mama's name was Goff. Mother taught us children to call them Uncle and I loved Uncle Henry as my uncle. He seems to think the world of me. ..... He always stood for the Williamson's side and they seem to think a lot of him. ...."

Azariah and Henry weren't his "legitimate" kids or the cousins wouldn't be telling each other so hush-hush about it. He was certainly never married to a Goff. So, what was his motive for raising them? It does seem like he thought they were his (back in the days before DNA testing).

I imagine if one could find a direct legitimate male Williamson descendent and a direct legitimate male Goff descendent and compared the y-dna for a family match we might could be a bit surer of the relationship.