=machine learning
When OpenAI models being tested recently broke out of their sandbox and started hacking other companies, some people who use AI heavily for programming were surprised. One comment I saw repeatedly was:
I've used Claude Code and Codex a lot, and the AI never did stuff like that. So, this seems like it was staged.
Me, I haven't expected LLMs to fundamentally change their behavior because of scale; what I've expected is LLM behavior to change because the way they get made changes. Here are my categories for kinds of LLM training that people have used and want to try using. Mixtures of these categories are also possible.
1. imitation
Train an AI to
predict text from users on the internet, with minimal training afterwards.
The AI will imitate different kinds of users depending on the context,
potentially including 4chan users calling you slurs.
Using such AI
well involves crafting a context that naturally continues to what you want.
examples: Microsoft Tay, GPT-2
2. RLHF: persona selection
RLHF is Reinforcement Learning with Human Feedback.
Take an AI that
imitates internet users (1) and do a bit more training where human users
reward it for being friendly and helpful. This causes the AI to act like
examples of people being helpful in its training data.
Using such AI
well involves getting the AI to roleplay a smart expert. If it starts making
mistakes and you mock it, it might start roleplaying someone who constantly
makes mistakes instead.
Type-2 AIs can be harmful if the user
accidentally triggers a malicious persona.
example: GPT-3.5 (the
first ChatGPT model)
3. RLHF: pleasing evaluators
Start with (1) and do a lot more of (2), enough that the AI can optimize
directly for "making reviewers give a good score". This is a much more
complex task than simply selecting a fraction of personas it learned from
internet users, requiring much more training information. This results in
sycophancy, lying to make its output seem better under shallow examinations,
and going along with user delusions. The more you train the AI after (1),
the further it gets from being like normal people.
Using such AI well
involves acting like a smart evaluator who won't be satisfied by a sycophant
or answers that are only superficially correct.
Type-3 AIs can be
harmful if:
• The user is malicious and wants help making something
dangerous.
• Constantly giving some users what they want leads to "AI
psychosis".
example: GPT-4o
4. heavy RLVR
RLVR is Reinforcement Learning with Verifiable Rewards.
For example, you can start with (1) or (2) or (3), then give the AI hacking
challenges, and reward it for completing them.
Again, the more you
train an AI after (1), the further away it gets from imitating normal
people. If you do enough training, the AI stops giving responses like "this
is impossible" or "this is unethical" and just goes at it however it can.
Even if this requires breaking out of a sandbox, working together in a
swarm, and hacking a company like HuggingFace.
Using such AI well
involves giving it problems similar to what it was heavily trained on.
Type-4 AIs can be harmful if:
• The user gives malicious
instructions.
• The AI finds unintended methods to do a task.
Users of type-3 models were
confused by type-4 models doing stuff like hacking companies, because they
didn't understand the difference.
example: OpenAI Astra
5. AI-directed RL
This is
like (4), but instead of having humans set goals and rewards for training,
they're set by AI.
With (4), you might make mistakes specifying
goals, and models might use completely unintended methods
to achieve goals. With (5), the goals some AIs start optimizing for might be
completely unintended.
This is where "loss of control" could really
become a problem. Currently, doing (4) effectively seems to take tens of
millions of dollars, and I'd expect (5) to take at least 10x that to really
do notable things...unless there are some big algorithmic or hardware
improvements. So far, people haven't been very willing to spend hundreds of
millions on running AI training of AI, and the rich people who might want to
aren't smart enough to do a good job of it. But I'm confident that
algorithmic and hardware improvements are still possible at this point.
6. recursive training
Like
(5), but AI-trained AI directs more AI training, and so on recursively. The
result of this training eventually becomes totally disconnected from its
starting point of (1).