James was the younger son of Joseph Andrews of Shaw House in Berkshire, by his second wife, Elizabeth, the daughter of John Pettit of St Botolph's parish at Aldgate in the City of London. The oft-quoted statement that he was born in Newbury seems unlikely. The family were originally from Hampstead in Middlesex and his father did not purchase Shaw until he was fourteen years old. James was privately educated by Rev. Matthews of Shaw.

Andrews' father died in 1753, but he remained at Shaw with his elder half-brother, Joseph Andrews. The two were always very close. From the date of the original calling out of the Royal Berkshire Militia in 1757, he joined up as a lieutenant (his brother was captain) and served there until it was disembodied five years later. He then exchanged arms for the law, using the money from his lucrative London practice to finance the erection of a fine Strawberry Hill mansion, to designs by John Chute, on land he had purchased at Donnington, not far from his brother's estate.  The house, known as Donnington Grove (or sometimes Chaucer's Grove, after Thomas Chaucer of Donnington Castle), was completed in 1772. Andrews was great friends with Horace Walpole, so he had obviously been inspired by the latter's home, Strawberry Hill, in Twickenham; but the construction was expensive and, after only eleven years, he sold up to Lord North's private secretary, William Brummell.

Both James and his brother, Sir Joseph, who had been made a baronet in 1766, took a great interest in the care of the poor. In 1788, James published 'An Appeal to the Humane on behalf of Climbing Boys employed by the Chimney Sweepers' and Sir Joseph chaired the committee which pushed a bill through parliament to improve their conditions. The Bill was passed but ultimately failed to change anything. Later, Sir Joseph almost certainly had a great influence on the resolutions of the Justices' Meeting held at the Pelican Inn on his estate in 1795, which led to the adoption of the 'Speenhamland System' of poor relief across much of the country. 

James settled at Brompton Row in South Kensington and his house became famous as the meeting place for all of literary society. He had published his first work, 'The Savages of Europe,' a translation from the French (of Messrs. Lesuire and Louvel) in 1764 - a grotesque satire on the English. However, history was his real passion and he became a noted historian and antiquary. He took an interest in local West Berkshire history, especially at Shaw House, where he named the Cromwell and King Charles Rooms and popularized the story of the Charles I Shooting during the Second Battle of Newbury. He is best known for publishing a number of works on more wide-ranging English history, including 'Anecdotes, ancient and modern' (1789), an amusing collection of gossip from old books, gleaned from his own extensive library, as well as his brother's at Shaw; 'History of Great Britain from death of Henry VIII to accession of James VI of Scotland' (1796) intended as a continuation of Dr. Henry's History of Great Britain ; and 'History of Great Britain connected with the Chronology of Europe from Caesar's invasion to accession of Edward VI' (1794/5). Both histories, though long since superseded, contained much curious information from ancient literature. In 1798, Andrews joined the poet laureate, Henry James Pye of Faringdon House, in a five-act tragedy from the German, called the ‘Inquisitor.’ He also contributed many topographical papers to the ‘Archæologia’ and to the ‘Gentleman's Magazine.’

In his legal career, Andrews ultimately become, in 1792 and remained until his death, one of the magistrates at the police court in Queen Square, Westminster. He had also been Deputy-Lieutenant of Berkshire in 1770. His wife, Anne, was the eldest daughter of the Rev. Thomas Penrose, the Rector of Newbury, and his wife, Anne, the daughter of Lawrence Head, the Mayor of Newbury; and a sister of the Rev. Thomas Penrose Junior, whose poetical pieces, which Andrews edited 1781, are included in several old collections of English poetry. The were married around 1765, at which time both had their portraits painted by Sir Joshua Reynolds. Anne Andrews died in October 1785, and James Pettit at Brompton on 6th August 1797. Both are buried in Hampstead Church in Middlesex. Their eldest son, Joseph, succeeded his uncle both as 2nd baronet and to the Shaw House estates in 1800