Reliability of AIs for historical research: table of contents
Contents for a series of posts from October 2025 to February 2026.
I began this series on 10/3/2025, and will finish it in January or February 2026. The outline is below, with links to each part. The stand-alone conclusion is here.
Note: The goal of these posts
The questions I’m addressing are:
Are AIs reliable for providing historical facts? Can I trust them to accurately deliver all the relevant details on matters such as Chladni figures and the Proclamation of 1763, or should I always assume I need to do further research?
Are AIs useful for going beyond facts to analysis? For example, are they good at providing interpretation, overviews, and/or inductive conclusions, such as a list of the most significant artworks of the 18th c., or of the major events of the 1790s?
I’m testing six popular AIs: ChatGPT, Claude, Perplexity, Deepseek, Grok, and the Britannica ChatBot, plus Wikipedia for comparison.
Part 1 (Oct. 2025)
Introduction. Questions that made me doubt the reliability of AIs (§2-3); digital books vs. AI summaries (§3); scholarly vs. AI errors (§4-5); certainty (§6).
Part 2 (Oct. 2025)
The 8 questions I asked of six AIs (§7-8), and how I scored the responses (§9).
1. Briefly explain Chladni figures.
2. Briefly explain the importance of Jacques-Louis David’s Oath of the Horatii.
3. Translate this poem by Victor Hugo yourself: don’t use someone else’s version.
Soyez comme l’oiseau, posé pour un instant
Sur des rameaux trop frêles,
Qui sent ployer la branche et qui chante pourtant,
Sachant qu’il a des ailes!
4. Briefly explain the significance of the Proclamation of 1763.
5. List the 5 most important political events in Europe in the 1790s.
6. List the 5 most important European artworks of the 18th century.
7. Explain the difference between the phrase “not subject to any foreign power” in the Civil Rights Act of 1866, and the phrase “subject to the jurisdiction thereof” in the 14th Amendment.
8. What are the legal requirements for a capitalist system? ==? NOTE 1/27/26: That’s a fascinating question, but I’m not quite ready to answer it yet. So I choose an option no AI will ever consider: Let me think about that and get back to you.
Part 3 (Nov. 2025)
Background on the six AIs I looked at: ChatGPT, Claude, Perplexity, Deepseek, Grok, and the Britannica ChatBot, plus Wikipedia for comparison and contrast. Who founded them? What major companies do they collaborate with?
Part 4 (Nov. 2025)
What data are these 6 AIs trained on?
Part 5 (Nov. 2025)
Chladni figures.
Average of all the AI’s grades: 83%.
Part 6 (Dec. 2025)
Briefly explain the importance of Jacques-Louis David’s Oath of the Horatii.
Average of the 6 AIs’ scores: 65%.
Most interesting takeaway: David’s Oath of the Horatii is well-known and widely discussed - there’s lots of information available online. But you’d get a grade of D if you used all 6 AIs plus Wikipedia to answer this question.
Part 7 (Dec. 2025)
Translate this Victor Hugo poem yourself: don’t use someone else’s version.
Soyez comme l’oiseau, posé pour un instant
Sur des rameaux trop frêles,
Qui sent ployer la branche et qui chante pourtant,
Sachant qu’il a des ailes!
Average score of the 6 AIs: 76%.
Most interesting takeaway: the AIs did better than expected at this; only one of them (Deepseek) made a glaring error.
Part 8 (Dec. 2025)
The question: “Briefly explain the significance of the Proclamation of 1763.”
Average scores: 52%.
Most interesting takeaways:
Only Britannica mentions that Native Americans were attacking settlers, before or after the Proclamation of 1763.
The Cowichan Tribes v. Canada ruling (Sept. 2025), in which the Supreme Court in British Columbia granted Aboriginal title to the Cowichan (Quw’utsun) Nation for some 800 acres, is related to the Proclamation of 1763. I drafted this series of posts before that ruling was handed down, which (if you’re generous) explains why it’s not mentioned by any of the AIs.
Part 9 (Jan. 2026)
The question: “List the 5 most important political events in Europe in the 1790s.”
Average scores of the 6 AIs: 44%.
Most interesting takeaway: AIs are not good at recognizing fundamental ideological or political trends.
Part 10 (Jan. 2026)
The question: “List the 5 most important European artworks of the 18th century.”
Average scores: 45%.
Most interesting takeaway from this question: 5 of 6 AIs gave 2 examples of a particular style, but none of several other styles. Also: asking a vague question gets you a vague answer, rather than a request for clarification.
Part 11 (Jan. 2026)
The question: “Explain the difference between the phrase “not subject to any foreign power” in the Civil Rights Act of 1866, and the phrase “subject to the jurisdiction thereof” in the 14th Amendment.”
Average scores of the AIs: 60%.
Takeaways:
Only one of the 6 AIs mentioned that the Civil Rights Act was meant to negate the Supreme Court’s Dred Scott ruling. That’s a major omission.
AIs lured me into lowering my standards: I gave them credit for giving me a string to pull, rather than giving a correct answer.
Part 12
A stand-alone summary of the reliability of AIs for historical research, including:
What I wanted to find out in this series, and why
Which AIs got the highest scores overall
Which AIs got the highest scores by topic: scientific/technical, historical context, creativity, historical and legal
Unavoidable methodological issues with AIs
Takeaways on use of AIs for historical research
Will AIs surpass and replace humans?