I'm not sure if the information I have is related to your James Williams, but my gggrandfather Samuel Cole Williams came down from the Sugar Grove/Rye Valley area of Virginia with some of his relatives. One was a James Williams who might have had the James W. Williams you speak of.
My info is as follows: hope it helps:
Children of Richard and Keziah Williams: all born in Smyth County, VA: and at least five of whom came to Tennessee in the early 1800’s: Samuel, Mary, James, Uriah, and Sampson.
Martha Williams (3/11/1793- ) m. John McClure
Elizabeth Williams (4/1/1794-7/2/1861) m. Aaron Blankenship
Samuel Williams (11/9/1795-4/19/1868) m. Emily Hubble (Hubbell)
Mary Williams (9/26/1797- ) m. Enoch Keen and lived on Little Tennessee 2 miles north of Morganton, TN.
James Williams (8/14/1799- ) m. and moved to Blount County.
Uria or Uriah Williams (4/3/1801- ) m. David Cress and moved to Johnson County, TN. This line has important family Bible.
Sally Williams (2/22/1803- ) m. Richard Williams (son of Joseph ; cousins)
Elisha Williams (2/27/1805- m. 3/4/1824
Sampson Williams (4/16/1807- ) moved to Tennessee same time as Samuel or before.
Freelove Williams (2/2/1809- ) m. Cress
Malinda Williams (5/28/1812-4/1869) m. Aaron Shupe. She is buried in Morgan
Cemetery, Sugar Grove, VA. Her stone
reads: “Don’t weep for me my children.
I am not dead but sleeping here.”
Lucinda Williams (12/21/1813-4/9/1857) m. George Washington Quillen
Hugh L. Williams (1/29/1816- ) Hugh might have moved to TN also, since there are references to a Hugh Williams in Blount County