Would skim milk prevent traditional fudge (no condensed milk) from setting up? I know it achieved the right temp and stirred it for 20 minutes.

I'm trying to find a good reference for you. I think it doesn't work because you need more fat and less water. When you cook the mixture to the soft ball stage, what you've done is cause a certain amount to the water to evaporate, leaving behind mostly fat and sugar with enough water to make it soft, and to enhance the crystallization once the fudge mixture cools down. By using skim milk, you're changing the ratio of water to fat. There are recipes for fudge that don't use any milk at all, but they do use chocolate and butter. I think if you're trying to make a low fat fudge, that's probably an oxymoron, but if you just don't have condensed milk, you could try a different recipe like this one: http://www.seriouseats.com/recipes/2014/02/baileys-fudge-squares.html