The founder of the family was an admiral, supposedly a cousin of William the Conqueror. Traditionally King Canute (1017-36) built the first castle on the site of Raby and this great castle which still looks like a fourteenth century fortress from outside, became the seat of the Nevilles. Their military base was Brancepeth Castle however, of which little remains from Neville times. Robert Neville, nicknamed 'The Peacock' for his arrogance and love of finery, slew Richard Fitzmarmaduke on Framwellgate Bridge in Durham in 1318 and was in turn killed by the Black Douglas near Wooler on 6 June 1319. He is buried in Brancepeth church, alongside the second and third Neville Earls of Westmorland.

In 1346 Ralph Neville (1299?-1367), brother of Robert, was victorious at the battle of Neville's Cross near Durham over the Scottish King David II, who was taken prisoner. This feat allowed him to be the first layman ever to be buried in Durham Cathedral; his son John Neville (d. 1388), a famous warrior in Edward III's French wars, became Lord High Admiral of England and began to fortify Raby in 1397. He is also buried in the Cathedral. John Neville is responsible for the magnificent Neville screen behind the altar 1380 - still a wonder despite lacking its original bright colouring and 107 alabaster figures. His son Ralph (1364-1425) was created the first Earl of Westmorland in 1397 and his great alabaster tomb may be seen in Staindrop church. He assisted at Henry IV's coronation and was Earl Marshal of England in 1399. He fought alongside Henry V in France and his was the prudent voice wishing for ten thousand more at Agincourt in Shakespeare's play. He was one of the executors of Henry V's will. His wife Joan Beaufort gives her name to Joan's Tower at Raby today.

Among their children was Robert Neville (1404-57) bishop of Salisbury and Durham, where he was visited by Henry VI in 1448. They were also the parents of Cicely the 'Rose of Raby', who married Richard Plantagenet, Duke of York, one of the leading figures in the Wars of the Roses. She became mother of two English kings, Edward IV and Richard III, and grandmother of Elizabeth of York, Henry VII's queen; thus she is a direct ancestress of the present royal family.

Edward IV was a very handsome man, inheriting his smooth good looks from his mother Cicely Neville, but he was intelligent too and very good at public relations, a fine dancer and fond of women. It was the Tudor propagandists who made his brother Richard out to be a misshapen, crook-backed monster. He was also tall and good-looking, though thinner than the well-built Edward. He was Cicely's eleventh child, and the Countess of Desmond, who lived to be over 100, told Sir Walter Raleigh that she had often danced with Richard and that he was the handsomest man at court, with the exception of his brother Edward. Richard virtually ruled the North for his brother from 1471 and appears to have been popular. His memory is celebrated at Middleham in North Yorkshire. When he proclaimed himself king in 1483, he began to place his northern adherents in positions of authority, the last time the North has managed to do this.

Richard Neville, Earl of Warwick (1428-71), the great 'kingmaker' of the Wars of the Roses, was a nephew of Cicely Neville. The Nevilles had become the most powerful family in the country, but the sixth Earl (1543-1601) supported the formidable but ultimately disastrous rising of the Northern Earls against Elizabeth in 1569. With the Earl of Northumberland, he marched to release Mary Queen of Scots at Tutbury, but was thwarted and fled abroad on the collapse of the rebellion. Raby, Brancepeth and all the Neville estates were forfeited to the crown. As Sir Cuthbert Sharp writes:

'Thus terminated an enterprise, begun without foresight, conducted without energy, and ending in dastardly and inglorious flight: entailing on the families of those concerned lasting misery; and inflicting on the leaders attainder, proscription and death.'

Eventually Raby passed into the hands of the Vane family whose family tombs are also in Staindrop church. In the twentieth century, a descendant of the first Earl of Westmorland married the ninth Lord Barnard and so Neville blood has returned to Raby.