So I think there may be two John P Bryants with sons name John P Bryant... ages are similar though the junior John P Bryant's are of different enough ages. So the census you reference:
http://search.ancestry.com/cgi-bin/sse.dll?db=1880usfedcen&a...Has a John Bryant born 1807 in Georgia, married to Amanda as you say... and a son John P Bryant born around 1857.
The thing is John P Bryant who married Virginia Cooper looks like he died before 1860... here's census progression:
http://search.ancestry.com/cgi-bin/sse.dll?db=1850usfedcenan...1850... he and Virginia are in Georgia in Gilmer County listed with their children, including a son named John Bryant born in a 1849 (only 1 year old).. This is before the John P Bryant son of Amanda was born.
In 1860, Virginia (incorrectly interpreted as Berginia, but you can look at the source) is now the head of the household, you again see a John Bryant child born 1849. This is in Pickens County Georgia. Pickens was formed from part of Gilmer in 1853, if you look at the Georgia Land Lottery of 1832 and the specifc plat awarded to John P Bryant you can see it's on the border of Gilmer that become Pickens (in fact I think it's smack in the now Jasper city area as far as I can tell). So they never moved, the county was created and formed around them
http://search.ancestry.com/cgi-bin/sse.dll?db=1860usfedcenan...In 1870 It looks like Virginia died and all the children are either married or split up to different households.
You can see a John Bryant living in Gilmer County married to a Mary Meeler with a son less than a year named James M (born 1869). Note that he's living next to his in-law Meeler family and in hat household next door is his younger brother Elias Bryant (listed as the youngest child in Virginia's household in the 1860 census, Elias was my ancestor). Again this John Bryant matching the trail as being born in Georgia in 1849:
http://search.ancestry.com/cgi-bin/sse.dll?db=1870usfedcen&a...In 1880 You can see that John Bryant listed as John P Bryant, married to a "Sarah E". He's listed as being born in 1850 (within 1 year of the trail we're following) and he has a son James M born 1870 (within 1 year of the James M above). It looks like he remarried but very clearly the same John P Bryant that's a son of John P Bryant and Virginia Cooper, he's stayed in the same two counties the entire time.
http://search.ancestry.com/cgi-bin/sse.dll?db=1880usfedcen&a...So it seems certain that there was John P Bryant son of a John Bryant and an Amanda born in 1857 who went with his family to Tennessee. Then there's a different one born to a John P Bryant and Virginia Cooper born in 1849 that stayed in Georgia.
Now the John Bryant senior could theoretically be the same person, left his first family and married a second one... though I would guess they were different ones without further evidence.