I will begin with what I know for actual fact:
Our father was William Jackson Rylee the 2st born 1905 in Mississippi.

He was the son of William Jackson the 1st born 1871 in Mississippi.

Wm.J., the 1st was the son of Andrew Jackson Rylee and his first wife Artemis Hargrove/Margrove

Andrew Jackson Rylee was born in Georgia in 1837
.
On the 1870 census for Miississippi > Marshall CO.> Holly Springs:
Andrew Jackson, born GA, is shown with Artemis, age 26 born MS
AND
Harriet, age 26 born GA
Decie age 14 born GA.
I assumed that Harriet and Decie were sisters.

A search of Georgia census records for 1860 confirms this:
Militia District 2, Banks, GA Gillsville P.O..
Matilda Riley age 43 born GA Head
Andrew J. Riley age 22 Farmer born GA.
Harriet Riley age 17 Spinstress GA
Thomas Riley age 15 GA
Deacjann Riley age 4 GA

(Thus a match with 1870 MS census. We also know that Andrew Jackson Rylee was in the Confederate Army and served under J.E.B.Stuart. I have found records that verify the Confederate service was with Cobb’s Legion, GA. Company C as a Private.and that he was with “Lee at Appomattox”
My brother says he remembers his great grandfather Andrew also had a brother Thomas. I have found Thomas on censuses for Holly Springs, MS area and all censuses say he was born in GA. I have also found marriage and death for Harriet
There were early family stories that somehow tied in the Mississippi move of Andrew and Thomas to a duel????.)

Now here is where I have the proverbial wall. As you know there were many Rileys/Ryleys/Rylees in early GA. I have seen many posts concerning the Rev. James Riley/Rylee, Sr and his Will and named children. There was a John.
This John is the closest match I can find to Matilda Rylee of 1860 census---but it is not a perfect match:
1850 Georgia > Hall CO.
John Riley 66 VA
M f 30
M f 11
A. m 10
H f 7
B. m 6
F. m 5
Of course there is no listing of a “T” for Thomas—unless the “F” age 5 is really a “T”. Then who is the “B”?. The “M” age 11 could have been married or deceased by 1860. Those census takers who only listed initials certainly did none of us any favors!
Some sources say John had a son William. Others also say he was married more than once.

Anyway no other census data comes closer. Can anyone help?