If using the whole milk version how will it impact sweetness? Also swapping coolwhip out for homemade whipped cream?

Condensed milk has a lot of sugar added to it. Using whole milk instead will reduce sweetness of that part of the filling by about 80%.
Using whipped cream instead of a manufactured oil-sugar topping like Cool Whip will give real dairy flavor, richness from the dairy fat and - depending on whether you add sugar to it and how much - probably less sweetness. It is also what Bill Smith used in his original version of Atlantic Beach Pie.
Both the original pie and this recipe variation are well balanced and your substitutions wont disturb that balance...just push it more toward your tastes.