Kinsey quotes "This is the way the world ends. This is the way the world ends. This is the way the world ends. Not with a bang but a whimper" from the poem 'The Hollow Men' by T.S. Eliot.
The bizarre show that Roger and Jane put on is a "blackface" number. Blackface, in which a white (or, more rarely, a black) performer would paint his or her face black and sing and dance in a way that was stereotypically associated with African Americans. It was a hugely popular performance style during the nineteenth century and into the first part of the twentieth. However, during the time this episode is set (1963), it was starting to fall out of fashion somewhat, especially as awareness of racial inequalities and stereotypes rose among white Americans.
The Drapers live in Ossining, New York on the Hudson River. It's infamously known for the Ossining NY State Correctional Facility, known in locally as Sing Sing. The expression is used often in old movies such as "Breakfast at Tiffany's" and "Angels With Dirty Faces". Another NY colloquialism referring to Sing Sing is "being sent up the river" which meant going to jail.
"My Old Kentucky Home" is the state song of Kentucky, published by Stephen Foster in 1853.
While working over the weekend, Paul Kinsey uses a paddle to hit a ball into a trash bin. This paddle is mentioned as being Harry Crane's. The paddle is that of the frat Alpha Delta Phi. This is the fraternity that Crane mentioned being a member of in Season 1, Episode 9 "Shoot".