Brown was the grandson of the medical theorist John Brown, founder of the Brunonian system of medicine. His great grandfather was a Scottish labourer. His father Ford Brown served as a purser in the Royal Navy, including a period serving under Sir Isaac Coffin and a period on HMS Arethusa. He left the Navy after the end of the Napoleonic Wars.
In 1818, Ford Brown married Caroline Madox, of an old Kentish family, from which his middle name was taken.[1] Brown's parents had limited financial resources, and they moved to Calais to seek cheaper lodgings, where their daughter Elizabeth Coffin was born in 1819 and their son Ford Madox Brown in 1821.
Brown's education was limited, as the family frequently moved between lodgings in the Pas-de-Calais and relatives in Kent, but he showed artistic talent in copying of old master prints. His father initially sought a naval career for his son, writing to his former captain Sir Isaac Coffin. The family moved to Bruges in 1835 so Brown could study at the academy under Albert Gregorius. Brown moved to Ghent in 1836 to continue his studies under Pieter van Hanselaere. He moved to Antwerp in 1837 to study under Gustaf Wappers. He continued to study in Antwerp after his mother's death in 1839. His sister died in 1840, and then his father in 1842.
 1936.116, Ford Madox Brown.1200x1200.jpg)
 1925.94, Ford Madox Brown.1200x1200.jpg)
