Raymond married Doris Scarborough (July 8, 1926- Dec 5, 2014), in 1942. He was twenty-two and she was fifteen. Doris was born to John Calvin Scarborough Sr. (January 1, 1879- April 13, 1959) and Annie Elvie Mazingo (October 18, 1880- December 21, 1951). Doris had six siblings. Doris attended elementary school to the 4th grade. They had one daughter, Ella Katherine Brown Partridge.
Prior to 1942, Raymond lived with Ella Mae and Isaac in Leake, MS. According to the 1940 Census, Raymond worked on roads in the area as an occupation and completed school to the fifth grade.