Stanfield was born in Sunderland, the son of the actor and author James Field Stanfield (d. 1824). James had written Observations on a Guinea Voyage (1788) and a poem The Guinea Voyage (1789) based on his own experience in the slave trade, when he was one of three survivors of a horrific voyage. He named his son Clarkson after the great anti-slave trade agitator.
Clarkson went to sea as a boy where he was encouraged by the writer, Captain Marryat, to take up painting. He was press-ganged into the navy in 1812, but left in 1818 and became a scene-painter in London. He painted a drop-scene for Dickens and exhibited landscapes and marine subjects, like the wreck of the Spanish Armada and the aftermath of Trafalgar, at the Royal Academy. It was said that his seascapes were so realistic that one became sea-sick looking at them. There is also evidence that Turner regarded Clarkson as a rival who needed to be outshone. Ruskin called Stanfield 'the noblest master of cloud-form of all our artists.'