Would it make a big difference to the cake to use sour cream instead of yogurt?

Would probably taste better, richer. You might not even need the oil. Sour cream has almost 6 times the amount of fat as whole milk yogurt.

I haven't tried it with sour cream, but I bet that would work wonderfully! My guess is that it might be a slightly wetter batter, and it's possible the amount of oil could need some tweaking. In this case I went with yogurt because I like the flavor, and I keep it on hand.