Hi,
The 1850 census for Milo only lists one BROWN family.
It is Horace & his wife Hannah. They have 8 children ages 19 down to 3 years. They do not have a James.
The listings for Evergreen Cemetery in Milo (oldest 1/4) has BROWNs , but no James.
1880 census shows :
James BROWN, Other,sex:M, Single, White, age:68, born:ME
both parents born in Maine.
In this same house is a
Joseph BROWN, Other, sex:M,Widower,White, age:66, born: ME
both parents born in Maine.
They are living in the home of Moses C. Foss & his wife & child.
When was Philena born ? And/or what years was she married to which spouse? I can try & look them up in my census if you could give a little more information.