Yeast fail

Yeast dies at around 140F and regardless, that's not your issue - it's about gluten development and a high hydration ratio. Brioche is a very sticky dough and is easier to handle (not to mention more flavourful) if it sits overnight in the fridge. Whether the yeast is alive isn't going to affect how wet the dough is.

Having a look at the recipe, I'd suggest waiting until the dough is formed a bit more and gluten starts to develop before adding the butter in to the mixer as it kneads. (Fat will impede gluten development, so you want to get it going first.)

You can also let it rest for 20-30 minutes and then start to knead the dough. This is called autolyse and will let the flour fully hydrate first.

And if it doesn't shape into a tight smooth ball don't panic - an overnight rest will make it easier to work with. You can also use wet hands to make it easier to handle the dough before refrigerating (but lightly flour when shaping).

For what it's worth, if the yeast is fresh you don't need to heat up the milk - all that does is prove the yeast is alive. In pastry school we never warmed up the liquid for use in our breads, the friction from the mixer and eventually the proofing process is what started to wake the little guys up. If you're not sure, of course, best to check.

There seems to be some confusion here. I don't know if I have all the answers but this much I do know: Instant yeast doesn't require proofing, it's designed to be added to the dry ingredients. 120F-130F is the correct temperature for the liquid ingredients. I am unfamiliar with the recipe in question so this is only a guess: The flour is specified in cups which is problematic. One baker's cup is another's cup and a half. If the dough formed a sticky ball after adding more flour, it sounds like you did the right thing. I'm not sure why the question is tagged "Yeast Fail" so maybe I don't understand the problem…

120-130 is a bit hot -- I usually keep the liquid to 105-110 when proofing yeast.

But sadly, you never know when a yeast packet will have gone bad. That's why it's always good to proof yeast apart from the rest of the dough, so you can toss it and start over if necessary!

As for your dough, you might try proofing new yeast in some milk/water and adding that to the dough, then letting it rise again? Let us know what you try!