General John Ramsay was the son of the painter, Allan Ramsay. The artist's first wife, Anne Bayne, had died in childbirth and none of the children from the marriage reached adulthood. He remarried in 1752, eloping with the well-born Margaret Lindsay. The couple had three surviving children, Amelia, Charlotte and John. In 1782, after the death of his second wife, Allan Ramsay made his final trip to Italy, taking John out of Westminster School to accompany him. John went on to pursue a successful military career; later in life he became a keen collector of pictures and antiquities.