In a 2 or 3 quart saucepan, add 2 cups of milk, the lavender and the vanilla pod, split and seeds scraped into the milk. Warm over medium heat until the milk is hot but not boiling. Set aside to infuse for 30 minutes.
In a medium bowl, whisk together the sugar, cornstarch and salt. In a separate bowl, whisk together the remaining 2 cups of milk and the yolks. Add yolk mixture to cornstarch mixture, whisking thoroughly to eliminate any lumps.
Strain the warm milk mixture and discard the lavender and vanilla pod. Gradually whisk the vanilla and lavender infused milk into the rest of the ingredients before returning the pudding to the saucepan. Bring to a boil, and cook 2 minutes or until thick, whisking constantly.