From “Loudoun Co Death Register 1853-1896” by Frain and Hiatt:
SMALLWOOD, John Wesley, white male, d. 1 April 1869 in Clarke Co., of natural decay, unknown, parents: Polly Smallwood, b. Loudoun, stonemason, married, source: Thomas Smallwood, son, Bureau of Vital Statistics:1869:1
The 1860 Loudoun Census places him in Middleburg. It does not appear that he owned any land on which to have a family cemetery.
Why do you think there was a farm and family cemetery?