Midge dated him at Bryn Mawr, just before Joel.

S1E3 - (flashback) Joel scoops Midge out from under Palmer.

S1E7 - Abe brings Palmer Witherspoon up during an argument.

S5E8 - At a Bryn Mawr reunion, Midge gets an update on Palmer from her fellow former students.

JOEL: Hello, there. Remember me?
MIDGE: Vaguely.
JOEL: Hey. Joel Maisel.
PALMER: Palmer Witherspoon.
JOEL: Really?
MIDGE: Are you here with a date, Joel?
JOEL: Not yet. You want another drink?
PALMER: I'll get it. Same thing?
MIDGE: Amaretto sour.
JOEL: Yikes. We'll work on that.
MIDGE: Amaretto sour, please, Palmer.
JOEL: I'm good, Palmer, thanks. I thought we got Goebbels in '45.
MIDGE: Stop it.
JOEL: He's so white and shiny.
MIDGE: Palmer's a good guy.
JOEL: Yeah. Bar's ten feet away. Surprised he didn't get lost.
– S1E3

MIDGE: Palmer's coming back.
JOEL: Yeah, I saw him asking for directions.
– S1E3

ABE: Miriam doesn't have a type. My God, before Joel, you were with that ultra white Palmer Witherspoon.
MIDGE: Oh, that name you remember?
ABE: He was like a pole-vaulter from Triumph of the Will.
– S1E7

DANIELLA: Palmer Witherspoon is still brokenhearted.
TAMMY: And available. He and Tuffy just divorced.
MIDGE: Palmer Witherspoon had the personality of a shoehorn.
KIKI: He's a lead accountant at Deloitte now.
– S5E8

3. Because You Left

• List of Characters