Humanity’s Last Exam
Benchmarks are interesting.
Here’s the deep thought – at what point in the overall benchmark process will AI inject bias into the benchmark test? And to what end? Maybe not so deep a thought.
Humanity’s Last Exam has been bantered about extensively. Here’s a great place to catch up on it: Humanity’s Last Exam
My musings: check out the crazy difficulty of the questions:
So 2500 questions of this caliber of difficulty. The top AI models hit 20% accuracy in answering.
I would also note the Calibration Error, which is affirms that “ Given low performance on Humanity’s Last Exam, models should be calibrated, recognizing their uncertainty rather than confidently provide incorrect answers, indicative of confabulation/hallucination. To measure calibration, we prompt models to provide both an answer and their confidence from 0% to 100%. ” The better performing models – OpenAI o3 and o4-mini and Gemini 2.5 Pro – also have better Calbration Error numbers.
Perplexed-ity?
Perplexed-ity? I came across this blog post from Cloudfare: Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives I've read and heard a lot of positive things about Perplexity's Comet browser. I want to like them and I want to cheer...
Google Voice
I've been working a bit to update content on TQuist website. I had gotten a Google Voice number years ago but then never used it, and after a time it was recycled. I signed up for a new one today and the process is very, very smooth. One has to bear in mind that...
CMS sites revisited
Recent work research includes CMS review. I haven't looked into what is out there in a long time. The quick search hits showed lots of familiar faces and a couple new ones. I found this post informative:...
Congratulations Impact Makers
Impact Makers was awarded "Best for the World" by B Lab. Here's a Richmond Time's Dispatch mentioning of the award. I met Michael Pirron shortly after moving to Richmond in 2004, and have watched him methodically and conscientiously build Impact Makers into a...
DropBox Security
Oh, the line between security and convenience is harsh. While reading TechRepublic I found this interesting article on DropBox security. I love the convenience of cloud technologies, and use DropBox like lots of people, including the article author (Michael Kassner)....
GOOG perspective on links
Analytics is an area that I will be focusing quite a bit in the coming months. I read this article about linking and Penguin 2.0 changes on a website called "Search Engine Watch." There are many pieces to the content puzzle that websites have to face, and...
Google Apps v. Office 365
The TechRepublic newsletter is worth a scan every day. I found this gem recently and it's worth a read if you use, or might use in the future, either Google Apps or Microsoft Office. Here it is: Google Apps v. Office 365: Head-to-head comparison of features...