I agree with ChefJune. I might toss them in if they are small and not the slightest bit woody. But in the garden, I pinch and toss the blossoms as they form. Sort of on topic, Harold McGee, food science and chemistry guru, suggests not using stems or even the point where the leaf meets the stem to help ensure the pesto will be green rather than brown.

I wouldn't. and I wouldn't use stems, either. Pesto is a tricky condiment. Get the proportions off and what should be ethereal can have a woody, muddy taste. I wouldn't chance it.