I have had some success with this, but there is no methodology that works all of the time. Here are some general guidelines:

(1) Determine that the person does not match the Y-DNA haplotype of an early progenitor (ancestral haplotype) of the surname subgroup. If the surname project is located at FTDNA, this is a pretty good sign of an NPE at some time in a Colonial line. Some projects were housed at Ancestry, so it may be necessary to locate the project home page with an internet search. Some did not have their own site, though.

(2) If the person upgraded to 67 markers at FTDNA, it is very likely that he will have more than 1 match with likely surnames and a lot of singletons with random surnames.

(3) Look for paper trails for matching families at Ancestry, Rootsweb, Genforum or direct contact to determine if there is an overlap in time and place of ancestors.

(4) Take all three autosomal tests. You cannot know which database will have a productive match. My friend had no likely matches at 23andMe and Ancestry. Yet, he found 2 productive matches at FTDNA, which has a much smaller database.

(5) Keep at it. The more your experience, the better your intuition.