Preheat the oven to 350. Grease and line two 8-inch cake pans (at least 2-inches tall) with parchment paper.
In the bowl of a standing mixer fitted with a whisk attachment, whisk the egg whites until stiff (3-4 minutes). Transfer whites to a clean bowl and wipe out the mixer bowl.
In the now-clean standing mixer bowl, beat the butter with the paddle attachment until creamy (1-2 minutes). Add the sugar and beat again until light and fluffy (3-5 minutes). Scrape down the sides and add the egg yolks and vanilla, and beat again until combined.
Add 1/3 of the graham cracker mixture, and mix on low until combined. Add half of the milk, and then another 1/3 of the graham crackers, and then the milk, and then the remaining graham crackers, mixing until just combined after each addition.
Remove the mixing bowl from the stand. With a spatula, gently fold in the beaten egg whites until completely combined (no white streaks remaining). Evenly divide the cake batter between the two prepared pans, and bake for 20-25 minutes, until the cakes are golden, and a skewer inserted in the center comes out clean (or with just a tiny bit of crumb).
Move cakes to a wire rack, and let them cool in their pans for 30 minutes. Invert on to a wire rack, and let cool completely before frosting. (Cakes can also be wrapped in plastic wrap and refrigerated over night.)