I am proposing a family for your Samuel. Just as Samuel's birth year varies, so does Edward's (1843-1851) depending on the record, as does their sister, Elizabeth. Edward and Elizabeth were in Simcoe County in 1881 but they later moved to Toronto. It appears that Elizabeth never married. Records show that Edward & Elizabeth immigrated to Ontario in 1875.
Note that Samuel's daughters, Margaret & Mary, were not only on the 1901 census with their parents in Barrie, they were also picked up on the 1901 Census with their Aunt Elizabeth in Toronto.
https://familysearch.org/pal:/MM9.3.1/TH-1961-33749-4936-19?...1851 Census Ballinderry, Antrim, Ireland
William Abernathy Self M 55 married 1833 Shoemaker b. Antrim
Mary Ann Abernathy Wife F 42 Spinning Yarn
Elisabeth Abernathy Daughter F 10
Edward Abernathy Son M 8
Samuel Abernathy Son M 6
Name: Edward Abernethy
Birth Date: 1844
Age: 24
Spouse's Name: Agnes Nesbitt
Spouse's Birth Date: 1845
Spouse's Age: 23
Event Date: 07 Dec 1868
Event Place: Belfast, Antrim, Ireland
Father's Name: William Abernethy
Spouse's Father's Name: William Nesbitt
Name: Robert Abernethy
Gender: Male
Birth Date: 12 Aug 1874
Birthplace: Antrim, Ireland
Father's Name: Edward Abernethy
Mother's Name: Agnes Nesbitt
1881 Ontario Census 138 - Simcoe South, A3 - Tecumseth
Edward Abernethy 30 b. Ireland Farmer married
Agnes Abernethy 35 b. Ireland married
Annie Abernethy 12 b. Ireland
Elizabeth Abernethy 9 b. Ireland
Robert Abernethy 7 b. Ireland
William Abernethy 4 b. Ontario
Mary Ag. Abernethy 2 b. Ontario
Elizabeth Abernethy 27 b. Ireland
1891 Ontario Census 132 - York West County, St. Albans Ward
Edward Aberneathy 45 b. Ireland Gardner
Agnes Aberneathy 45 b. Ontario
Annie E Aberneathy 20 b. Ontario
Robert Aberneathy 17 b. Ontario
William E Aberneathy 13 b. Ontario
Mary A Aberneathy 11 b. Ontario
1901 Census of Canada District: ON YORK (West/Ouest) (#131)
15 6 146 Abernethy Elizabeth F Head S Sep 20 1850 50 b. Ireland, immigrated 1875, Seamstress single
15 7 146 Abernethy Mary F Niece S Jun 16 1880 20 b. Ontario
15 8 146 Abernethy Margrat F Niece S Apr 30 1883 17 b. Ontario
15 9 146 Pugley F Lodger W Apr 11 1857 44
15 10 146 Goodeham George M Lodger S Sep 16 1879 21
15 11 146 Goodeham Frank M Lodger S Oct 16 1884 16
1901 Census of Canada District: ON YORK (West/Ouest) (#131)
Subdistrict: Toronto (City/Cité) Ward/Quartier No. 6 G-19 Page 1
1 34 6 Abernathy Edward M Head M Nov 14 1847 53 b. Ireland, immigrated 1875, Gardner
1 35 6 Abernathy Agnes F Wife M May 16 1845 55 b. Ireland, immigrated 1875
1 36 6 Abernathy Robt M Son S Aug 12 1874 26 b. Ireland, immigrated 1875
1 37 6 Abernathy Wm E M Son S Jul 15 1877 23
1 38 6 b. Ontario Abernathy Mary A F Daughter S Jun 27 1879 21 b. Ontario
1 39 6 Abernathy Mary F Niece S Jun 1880 20 b. Ontario
Name: Margaret Abernethy
Age: 25
Birth Date: abt 1882
Birth Place: Cookstown
Marriage Date: 26 Jun 1907
Marriage County or District: Toronto, York
Father: Samuel Abernethy
Mother: Elizabeth McEachern
Spouse: Richard Orme Whitebread
Spouse's Age: 27
Spouse Birth Date: abt 1880
Spouse Birth Place: Craigvale Ont
Spouse Father: John Whitebread
Spouse Mother: Sarah Amelia Parker Whitebread
Name: Robert Abernethy
Age: 27
Birth Date: abt 1875
Birth Place: Belfast, Ireland
Marriage Date: 26 Jun 1902
Marriage County or District: York
Father: Edward Abernethy
Mother: Agnes Nesbitt
Spouse: Edith Josephine Elizabeth Clark
Spouse's Age: 27
Spouse Birth Date: abt 1875
Spouse Birth Place: Collingwood, Ontario
Spouse Father: George Wan Clark
Spouse Mother: Georgina Jane Bachelor Clark
Name: Edward Abernethy
Event Type: Death
Event Date: 24 Apr 1909
Event Place: Toronto, York, Ontario
Gender: Male
Age: 65
Birthplace: Ireland
Birth Year (Estimated): 1844
Name: Elizabeth Abernethy, single, died at 78 Landsowne Ave., Toronto
Gender: Female
Birth Year: abt 1840
Birth Place: Ireland
Age at Death: 77
Death Date: 29 Dec 1917
Death Place: York, Ontario, Canada
Burial: Park Lawn
I see you have a tree on Ancestry with these names on it so I am wondering why you placed a query.
Name: James Crothers
Spouse's Name: Rebecca Abernethy
Event Date: 22 Nov 1846
Event Place: 0769, Glenavy,Antrim,Ireland
Find a Grave - tombstone photo -
http://www.findagrave.com/cgi-bin/fg.cgi?page=gr&GRid=11...Name: Rebecca Abernathy Carruthers, wife of James
Birth Date: 1824
Birth Place: Ireland
Death Date: 6 Dec 1872
Death Place: Essa, Simcoe, Ontario, Canada
Cemetery: Christ Church-Saint Judes Anglican Cemetery
Burial or Cremation Place: Essa, Simcoe County, Ontario, Canada
Children: William John Carruthers
Susan Bell Carruthers