Fresh Thoughts #86: Beware Sprinkles of AI
- Newsletter
We are firmly in the grip of AI hype.
In August 2023, Goldman Sachs found one-third of S&P 500 companies mentioned AI on their earnings calls.
Is AI going to take over the world?
Beware Sprinkles of AI
Last week, the Hollywood writer's strike against AI - more specifically, Generative AI - ended.
The writers got what they wanted, including:
- AI can't be used to rewrite work
- Executives can't use AI to cut writers out of the process entirely
- A writer's work can't be used to train AI models
AI is not invading Hollywood's writing department - just yet.
I suspected this was going to be the inevitable conclusion.
You see, Generative AI is not creative.
It doesn't know how to play with context - or challenge preconceptions.
Generative AI doesn't comprehend what it is writing about.
Generative AI's trick is to guess what comes next.
At a foundational layer - the Large Language Models used in Generative AI tools don't even work with words.
They work with "tokens" - syllables, punctuation and spaces.
For example, given "in" as the starting point, the next token could be:
- in[space] - as in 👈
- in[period] - as in "Please come in."
- in[de] - as in the start of "independent"
- in[si] - as in the start of "insight" or "inside"
- ...
It's like auto-complete... on a massive scale.
Not a bad thing, but not a panacea.
And how does it learn what comes next?
Vast amounts of training.
ChatGPT - the most famous generative AI tool - was trained on roughly 300 million pages of text.
It knows the chance of insight vs. inside vs. independent because it has seen many, many examples of it before.
Can AI Analyse Phishing Emails?
I thought of this while investigating a suspected phishing attack last week.
At first glance, everything looked wrong.
It looked like a highly sophisticated attack.
The sending domain - spoofed.
The SPF and DKIM protection - spoofed.
The DKIM signature - dodgy.
The sender's address - spoofed.
But something - intuitively - felt off.
Where did it come from?
What is that odd header field - that I have never seen before?
What's going on?
To solve this - I had to gather more information and return to first principles.
Who owns the sending IP address?
What are the full DKIM and SPF settings?
And what is that odd header field?
I've never seen this before.
All of the well-known investigation tests pointed toward a sophisticated attack.
But within 10 minutes, I had worked it out.
A new management information system had been installed.
Only no one was informed about it.
---
There was no training data for this investigation.
It was novel and new.
AI could not change the investigation - based on a hunch.
And would have assumed it was an actual, highly sophisticated attack - setting off all the alarm bells in the process.
I am pro-automation...
I am pro-AI...
And AI certainly has a role to play in the future of cybersecurity.
But...
Beware sprinkles of AI.
Especially when they add more to the marketing and price than the ability to comprehend and mitigate the novel and new.
October 9, 20233 Minutes Read
