Have you looked at the 1870 census? I found this info.:
Horace Perry 57
Sarah Perry 40
George M Perry 11
Frank Perry 9
Willie Perry 7
They were in Conn. and Frank may have been born in 1861
I have Perrys in my family, but they never lived in Connecticut so I don't believe I'm connected with your family.