If I understand you correctly a male family member has Y DNA tested and shows his Y DNA matches Rose not Prince. Is this correct?

It is possible someone in this male line took their mother's maiden name of Prince when they were actually fathered by a Rose. This could be one explanation why you are matching both surnames through autosomal DNA which goes to all ancestors on all sides of your tree that you inherited. Y-DNA is much more specific on proving or disproving one male line. If your Y-DNA does not match your surname, In the DNA world they call this a NPE ( non-paternal event). There are several reasons one could have an NPE in their line. If your ancestor was born out of wedlock the law was your ancestor was to be given mother's maiden name not father's surname. If your ancestor was wanted by the law and happened to change his surname his Y DNA would not match what is on paper, an unk adoption, your ancestor was orphaned out and took on the surname of his caregiver. So many reasons.

The key with any DNA test is you must find paper to back up you matches. If your ancestor was born out of wedlock some states have what is called the bastardy bond records. A woman was supposed to call the father out in court so the state did not have to pay for the child. I know NC has this law, I believe also SC and VA. Not sure about other states. If your ancestor was in one of the states that has bastardy bonds you need to check county court minutes to find the bonds. Bondsman on the record is sometimes the father of the child and sometimes not. Some women would not reveal the actual father of the child and these bonds do not list name of the child so it makes it even harder proving the child in the bond might be yours.

Hope this helps.