James Doyle  was born in London on 22 October 1822, was the eldest son of John Doyle. Richard Doyle and Henry Edward Doyle were younger brothers. He was the uncle of Arthur Conan Doyle.

James was educated as a Roman Catholic. He inherited a portion of his father's artistic ability, and in early life studied drawing and painting. Among other works he executed a painting of Dr. Johnson reading the manuscript of the 'Vicar of Wakefield,' which was engraved and attained considerable popularity. The copyright of the picture realised £100. While comparatively young, however, Doyle abandoned the profession of an artist and devoted himself to historical studies. For his own edification he compiled a 'Chronicle of England' from B.C. 55 to A.D. 1485, which he adorned with numerous illustrations in colours. It received considerable praise from various persons to whom it was afterwards submitted, among others from the prince consort, and was well received by the public when published in 1864 (London, 12mo). Doyle's illustrations were engraved and printed in colours by Edmund Evans.

The great undertaking of Doyle's life, however, was his 'Official Baronage of England,' which included every rank of nobility except barons. The epithet 'official' in the title means not that Doyle's 'Baronage' was published 'by authority,' but that it gave an exhaustive list of the offices held by the peers of whom it treated. This compilation was at first designed especially to cover the period between the Norman Conquest and the Revolution of 1688, but it was afterwards brought down to 1885. It gave particulars, as complete as possible, of the succession, titles, offices, heraldic bearings, and personal appearance of each peer. This work was published in three quarto volumes in 1886, a large-paper edition, limited to two hundred copies, appearing somewhat earlier in 1885. It is a painstaking but unequal work. For the earlier portion, especially the Norman and Angevin period, Doyle relied too much on secondary authorities, and was not sufficiently critical. Greatly to his disappointment the book was not a financial success, and inflicted a heavy loss on the publishers. In 1886 he wrote the explanatory text for Richard Doyle's coloured cartoons, entitled 'Scenes from English History.'