Hi,
Have you traced either of these men in England to see where they lived or when they left England? If one or both left previous to 1884 then can't be your man.
You might need to consider DNA testing.
I doubt there will be anything you could find now which could prove 100% who Polly's father was.
Even if one of the Richards had an illegitimate child before leaving England, how would you prove that child was Polly?
Kellie