First read the consequences of a person who has a dishonorable discharge from a Military Service.
The severity of his crime that warranted such an action may have made necessary a name change to have a better life.
Back then one could basically hide a past with just a name change but not today. He would not have needed any legal proceedings either. Since the creation of the SSA and SSNs were not required till mid 1930's I bet your grandfather just filed for his SSN number under the name Michael Joseph Denis and then moved began the process of other forms of ID and future employment with that name. So as you say he moved from NJ to NY probably to to do this.
I do not find the census record for him as a "child", but looking at your 2005 query and the information on other relatives I did find him in the 1930 census under the name Michal Danish age 26 listed as a step-son in the household of a John Balazsi (as transcribed by Ancestry). Column 30 asks if a Veteran of the US Armed Forces and his is blank where others did answer yes or no. Could be census taker didn't think to ask as he was too young for WWI. But there is not a "No" for sure.
When you say the "Navy" does not have any records do you mean the Navy Dept or the NARA that stores past Naval Records.
http://www.archives.gov/veterans/You should look for Navy JAG/Court records as the Navy may have purged his records from their possession or the NARA storage as he was no longer even considered a "Veteran" but the Naval Court Proceedings should still be archived.
Order a FOIA copy of his SSN application to see his parents names as he listed them and the name he filed under. The number should be the same no matter what name he used to for his application. That should give you a time line of a name change. I would think any service record would be under the name Danish. Also order his sister's SSN application to compare information. Since his mother remarried was there a divorce or was she a widow? Would her second marriage certificate give the name of first husband? There should be court records of a divorce, again to looking for names....any minor children should be listed with birth names as would any probate court records.
Have you seen the 1910 census for a family named Danish living in Passaic Ward 4 Passaic Co. NJ. Head is a Joe, wife Therasia and a son Mice, could be "Michael/Michal" was also a name change from birth name as he grew older. The surname change was later in life.