In a medium saucepan, heat the olive oil and smashed garlic over low. Warm the oil though, tossing occasionally and taking care not to fry the garlic, about 3 minutes. Add the orange peel, rosemary, red pepper flakes, and olives. Cook, stirring occasionally, until the garlic is just starting to brown at the edges, about 5 minutes. Remove from the heat.
Stir in the orange juice, mixing to combine. Set aside to cool. Serve slightly warm or allow to cool completely before transfering (rosemary sprigs, orange peel, and all) into an airtight container to store in the fridge. Will keep for 1 week.