---
title: "The Rarity Shibboleth"
description: "Explore how AI and LLMs excel at averaging data, but fall short on finding rare insights. Analysis of Microsoft's Azure growth and AI revenue trends reveals key limitations."
categories: ["AI"]
keywords: ["Microsoft Azure growth","AI revenue trends","large language models","LLMs","data insights","TF-IDF","prompt engineering","Magic Prompt","rare data points","venture capital"]
ai_summary: "Explores the limitations of LLMs in finding rare insights compared to their averaging capabilities, using Microsoft Azure as a case study."
date: 2024-09-09
lastmod: 2026-07-28
canonical_url: https://www.tomtunguz.com/tfidf-in-llms/
author: "Tomasz Tunguz"
---

Large language models are wonderful at ingesting large amounts of content & summarizing. Uploading an academic paper and I can pester it with an infinite list of questions & it will respond with equally infinite patience. 

In comparing the two most recent Microsoft earnings calls, Claude highlighted:

- faster than expected Azure growth (29% vs 27%) 
- AI contributing 8% of Azure revenue up from 7%
- higher CapEx spending & greater capacity constraints for data centers
- much better commercial bookings growth : 17% vs 13%

Excellent analysis.

[Benn Stancil](https://tomtunguz.com/select-average-moby-dick-limit-2/) described LLMs as great averagers of information. But what if I don't want the average? If I seek the data point two standard deviations out? If I'm channeling my inner Anthony Bourdain & I'm seeking the [fermented shark or cobra heart](https://www.mashed.com/231840/the-craziest-things-anthony-bourdain-ever-ate/#:~:text=Perhaps%20one%20of%20the%20most,actually%20considered%20an%20Icelandic%20delicacy.) within my query?

At Google, we ranked web pages & ads with many signals. An engineer taught me that of them found rarity of a word across a set of documents. 

For example, if there's a document containing 10 instances of the word [ghoti](https://tomtunguz.com/teaching-a-computer-to-ghoti/) across a collection of documents, where the average document has 0 ghotis, that document is likely the best to answer a search about fish.<sup>1</sup> 

I haven't found a way to goad an LLM to produce the rare result. Why did one user decide to use a particular piece of software when 10 others did not?  

Maybe I haven't learned how to prompt an LLM well. Ideogram launched a feature called [Magic Prompt](https://docs.ideogram.ai/using-ideogram/prompting-tips/using-magic-prompt) that expands a basic prompt into richer instructions that elicit better results, marrying the language of a user & a computer.

A user might write : *A boy with a dog in a park*.

Magic Prompt replies : *A heartwarming scene of a young boy playing with his happy, wagging dog in a lush, green park. The boy has a playful smile on his face while he tosses a bright yellow ball for his furry friend to fetch. The park is filled with beautiful flowers and trees, creating a serene atmosphere for this bonding moment between boy and dog.*

Magic Prompt surprised me in three ways. 

First, unlike Google search where terse queries tend to work very well, LLMs relish verbosity. Less Hemingway & more Faulkner. 

Second, maybe there's a prompt for rarity. If you know what it is, please share with the rarity shibboleth.

Third, I wonder what other prompt techniques are out there that have yet to be discovered to wrest the best from an LLM.

---
<sup>1</sup> This technique is called [TF-IDF](https://en.wikipedia.org/wiki/Tf%E2%80%93idf). It means Term Frequency Inverse Document Frequency.