Ferguson, the eldest son of the famous philosopher, Professor Adam Ferguson, met Sir Walter Scott when they were both students at the University of Edinburgh. They remained life-long friends; Scott said that Ferguson had 'the lightest and most airy temper with the best and kindliest disposition'. Following his retirement from the army in 1817, Ferguson was appointed Keeper of the Regalia of Scotland, which had recently been re-discovered with Scott’s assistance. Ferguson received a knighthood from George IV during the King’s visit to Edinburgh in 1822.