Place the potatoes and garlic in a large stock pot. Don’t worry about the garlic, you’ll be getting rid of it later. Fill pot with water 2 inches above the potatoes and salt the water. Bring to a boil over high heat. Reduce heat to medium and summer, about 10 minutes, until fork tender.
Meanwhile, heat olive oil over medium heat in a skillet. Cook bacon, stirring occasionally, until brown and crispy, about 10 minutes. Remove the skillet from heat, and stir in the onion, celery, and vinegar. Season with salt to taste.
Drain the potatoes and transfer them to a large bowl, throw away the garlic. Add the bacon mixture to the hot potatoes and toss. Gently fold in the mayo and mustard. Refrigerate until cold and serve.