Where can I buy lamb kidneys?

Small farms will sell any part of their meat, including heart, kidneys, tongue, marrow bones, and liver. Find out what small meat farms come to your local farmer's market, send them an e-mail to reserve some kidneys for you, and then pick them up on the next farmer's market day. I do this when I want marrow bones for soup.

White Oak Pastures in Bluffton, GA, sells them (and just about any part of the animals they raise, as they are a farm focusing on animal welfare and environmental sustainability). They list their products online and will ship, but I have no idea about the shipping costs. If you can find a local place near you, obviously that would be a better choice, but these folks are really great.

Good idea about a Halal butcher. I haven't been, but have heard good things about Honest Chops in the East Village - since they're a "whole animal" butcher, maybe they'd have it: http://honestchops.com/

I would check a halal meat market.

You will probably have to make a special request at a real butcher shop (Try Pino's on MacDougal just south of Houston) or at the Union Square Greenmarket on Saturdays at the Catskill Merino (lamb farm) stand. I'm sure either place can get them for you.

You will have to go to a real butcher for those. And it may require a special order. Some lamb parts e.g. lungs, can't be sold in the USA.