Spread the bacon in a soup pan and place over medium heat. Cook until browned and its fat has rendered. Drain. Discard all but 1 tablespoon of fat and put the pan back on the heat. Scrape the onion and celery into the pan and sauté until softened. Drop in the potatoes. Add enough water to cover, about 3 cups. Bring to a boil, then reduce the heat and simmer until the potatoes are tender, 15 minutes.
Take the pan off the heat and, using a masher, lightly crush the potatoes to thicken the chowder. Add the oysters and oyster liquor and season to taste with salt and cayenne pepper. Simmer until the oysters curl, 3 to 5 minutes.
Pour in the milk (use 2/ 3 cup if you prefer it less rich), bring to a boil, then shut off the heat. Crumble in the bacon and stir in the parsley.