Hello: George first appears in the 1870 census for Union County NJ, living with wife Johanna and a Sarah B Bryant who is 23 years old. In the 1880 census she is 30 years old, and identifies herself as George's sister. In the 1870 census there is a George L Bryant living with the family who is 80 years old (21 years older than your George), who might be the father. I could not find any previous census records for this group. There is also a Jennie E Bryant, age 18 in the 1870 census group, who is working as a school teacher. George the elder was listed as a railway conductor, and your George was listed as a farmer. Hope this helps.
Best wishes
Norm Dawson