Preheat the oven to 350 degrees. Throughly wash and dry the fruit. Slice the peaches into wedges and set aside.
Set a 9-ince cake pan on top of a burner set to low heat. Add 4 tablespoons of butter to the cake pan and allow to melt. Add the brown sugar and lemon verbena leaves and stir gently until the sugar has melted. Turn off the heat, but leave the pan on the burner.
In a medium bowl, add the flour, baking powder and salt and gently whisk by hand. In a large bowl, add the remaining butter and sugar and beat on medium speed until light and fluffy. Add the eggs, one at a time, beating well in between each addition. Add the vanilla. Turn the mixer to a low speed and add the dry mixture. Mix until just combined. The batter should be thick.
Carefully remove the lemon verbena leaves from the cake pan. Using a rubber spatula, smooth the brown sugar mixture so that the entire bottom of the cake pan is covered with a thin layer. Arrange the fruit in the baking pan, being sure to tightly pack the fruit (keep in mind that the fruit will shrink when cooked). Using an ice cream scoop, dollop the cake batter on top of the fruit. Once all cake batter is on top of the fruit, use a rubber spatula or icing tool to smooth the batter across the cake pan, ensuring that it is spread in an even layer. Bake for 40 minutes.
Remove the cake from the oven and allow to cool for only 1-2 minutes. Run a knife around the edges of the cake pan and place a plate over the cake. Very carefully flip the cake pan upside down. The cake should very easily slide out on the plate. Once the cake is on the plate and out of the pan, allow to cool completely so that the fruit continues to set.