It proves one side of a debate. Whether or not LLM weights contain "content." If a model trained on reading text embeddings can read model weights, then it implies that somewhat similar representations exist in the weights.
When a model recites Shakespeare verbatim what happened is that when training, shadows of that text were cast on those weights. And during inference, shadows of those weights are being cast onto output tokens. The text is in there. And because of different forms of gating like head selection, attention masking, attention in general, and relu, the text isn't even smeared over all of the weights. The gradient for a training example ends up cast on a smaller region of the model. So copyrighted material exists in that model within some representation. So much so that with a few transformations, astra can read the content.
It acts like a soft key-value lookup, not because we've trained it to behave like one, but because under the hood it is one.
Log in
Well that's insane.
It proves one side of a debate. Whether or not LLM weights contain "content." If a model trained on reading text embeddings can read model weights, then it implies that somewhat similar representations exist in the weights.
When a model recites Shakespeare verbatim what happened is that when training, shadows of that text were cast on those weights. And during inference, shadows of those weights are being cast onto output tokens. The text is in there. And because of different forms of gating like head selection, attention masking, attention in general, and relu, the text isn't even smeared over all of the weights. The gradient for a training example ends up cast on a smaller region of the model. So copyrighted material exists in that model within some representation. So much so that with a few transformations, astra can read the content.
It acts like a soft key-value lookup, not because we've trained it to behave like one, but because under the hood it is one.