Poverty. That is the simplest way to describe the constellation of factors that contributed to high infant mortality. Were the families you mention poor?
Poverty leads to malnutrition, and nutritional deficiencies lower the body's ability to ward off disease. Crowded and/or unsanitary conditions (also reflections of poverty) hasten the spread of disease and limit the chances of survival for those affected by diseases. Tuberculosis was the cause of more deaths in industrialized countries than any other disease during the 19th and early 20th centuries (see link). Cholera, diptheria, small pox, whopping cough, and dehydration due to diarrhea could prove deadly for any person but especially to compromised infants. Many children died from diseases we don't particularly fear nowadays, such as measles and chicken pox and influenza. Some infants died of malnutrition itself (see link in diskoverit's post above). You might be able to find out about specific conditions and specific epidemics in the exact years and places you are studying. There are many social histories and medical histories and social science studies that show that the wealthier your parents were, the better your chance of living to age five.
http://www.123helpme.com/view.asp?id=22339http://logicmgmt.com/1876/overview/medicine/diseases.htmhttp://ocp.hul.harvard.edu/contagion/cholera.htmlhttp://ocp.hul.harvard.edu/contagion/tuberculosis.html(the above remains true for millions of children today)