Thanks so much! I really appreciate the help, alot of info here to consider! I too saw this family in the 1851 census and wondered if this was the right family - James & Agnes is certainly promising. While there is no Jeremiah, there is a Thomas (Jeremiah's middle name) who is only a couple of years older than what I know Jeremiah to be. Was this common (using middle name?).
The main complication for me was that I have the marriage record of James and Agnes from 1837. It'd be unlikely that they'd have had at least 5 kids (John, Robert, Elanor, James, Jane) before being married in 1837. Maybe a second marriage for James?
Thanks, this is the best lead yet! Would love to track them back to Ireland and this is still promising ...