You'll find some info about the Monroe County Poor Farm at www.poorhousestory.com/poorhouses_in_kentucky.htm
Dory wasn't in the list of inhabitants in 1880. They did have a cemetery at the Poor House but stones overturned and no burial records.
The South Central Kentucky Cultural Center, www.kyculturalcenter.org/
and click on Library might have records.
The Monroe County Historical & Genealogical Society, www.facebook.com/MonroeHistoryMuseum/
Monroe County had courthouse fires 22 April 1863 and again in 1888 so I don't know what records survived. Successful Searching!