James George Oliver was born in 1842 as the son of Simon, Sr. and Mary (Wright) Oliver.

He was married to Martha Hawkins and lived in Lenawee County, Michigan.