What is the best butter substitution when baking cookies? Coconut oil? Applesauce? Margarine?

You can use half canola oil and half applesauce but the cookies will be cakey and not crisp. If that is okay it works great.

I have had success using coconut oil instead of butter, although in baking the cookies seem to come out flatter.

It depends on why you don't want to use butter. If it is for vegetarian reasons, any of these suggestions can work. If you are trying to minimize saturated fats, skip the coconut oil. If you want to minimize trans fats, margarine is not a good choice. Crisco has a mixture of oil and saturated fats but only a trace of trans fats (0.3%). Lard is actually more healthy than butter on the basis of saturated fats, but less so than shortening. But of course, since it is from animal sources, it is not for vegetarians.

Lard or Shortening.

I use margarine, preferably EarthBalance or another soy one. Sometimes Fleischman's unsalted. When I'm not using butter. Which happens sometimes.