Combine milk and lavender spigs in a small pot over low heat. Bring to a simmer and turn away from heat immediately. Cover and let cool. Strain milk through a fine mesh sieve, press lavender sprigs to extract as much flavor as you can. Set aside.
Preheat oven to 350 F. Butter a loaf pan and dust with flour. Set aside.
In a large bowl, combine cake flour, baking powder and salt. Whisk until well combined. Set aside.
In the bowl of a food processor with the dough blade attached, mix sugar, lemon and orange zest for a minute or until very fragrant. Add butter and mix again until light and fluffy. About 2 minutes. Add eggs, 1 at a time. With the food processor running on low speed, add the oil in a slow steady stream.
Alternate between flour mix and lavender milk, starting and ending with flour. Pour batter into pan and bake until cake springs back when pressed, or when inserted toothpick comes out clean. (40 to 50 minutes)
Allow cake to cool for 15 - 20 minutes. Run a knife around the edges, and flip pan on a clean kitchen towel. Transfer cake to a plate and allow to cool completely.