The family is on the 1880 Franklin Cty Ga census. This may be them in 1860:
1860 Habersham cty Ga'
740 740 Mies A Loftis 29 SC
Lithinia, 26 Ga
Frances C 6, Ga
Susan T Ga
741 741 James Loftis, 22 SC
Marth??? A 19 SC
739 739 Perry Loftis 59 SC
Polly An 62 SC
Hester Ann 39 SC
Anna M 31 SC
Nancy C 8 Ga
Andrew 15 Ga
Mary F 1/12 Ga

1850 Habersham Cty Ga
105 105 ?erry Loftis 51m
Mary 52
H L 24f
A M 18f
M A 16m
W E 13m
S P 11m
A J 8m

Perry Loftis is also on the 1840 Haberhsam Cty census.

One family tree posted at Ancestry.Com has Perry as son of James Loftis and Catherine Morris.