I would concentrate on the 1870 and 1880 census schedules. Did James George Smith marry? If so, look at the census schedules on either side of the marriage year and also in the county of marriage. Good luck.