If a recipe call for an 8x8 pan, how much should I increase the recipe for a 10x12 half tray?

You can also always reference cake pan volumes online (see http://www.joyofbaking.com/PanSizes.html for just one example) and then double or triple according to how much more volume you need.

The area of 8x8=64 The area of 10x12=120, so doubling the recipe should work just fine.