Skirving was born near Haddington, East Lothian. After being educated locally, he began work as a junior clerk at the Customs Office in Edinburgh. It is thought that Skirving also studied at the Trustees' Academy in the city at the same time, while painting miniatures at night. By the end of 1777 Skirving decided to move to London to work as a professional artist. After some success, he returned to Edinburgh in the mid 1780s before spending seven working in Rome from 1787.