Oh that is super interesting! I wonder if they track how often it succeeds in matching and replacing, I'd love to see those numbers in aggregate.
Total anecdote, but I worked on this for a bit for a research-level-code code editor (system paper to come soon, fingers crossed!) and found that basic find-and-replace was pretty brittle. I also had to be confident the source appears only once (not always the case for my use case), and there was a tradeoff of fuzziness of match / likelihood of perfectly correct source.
But yeah, diffs are super hard because the format requires far context and accurate mathematical computation.
Ultimately, the version of this that worked the best for me was a total hack:
Prefix every line of the code with L#### -- the line number. Ask for diffs to be the original text and the complete replacement text including the line number prefix on both original and replacement. Then, to apply, fuzzy match on both line number and context.
I suspect this worked as well as it did because it transmutes the math and computation problems into pattern-matching and copying problems, which LLMs are (still) much better at these days.
I suspect any other "hook" would work just as well, a comment with a nonce--and could serve as block boundaries to make changes more likely to be complete?
This is actually a very powerful pattern that everybody building with LLMs should pay attention to, especially when combined with structured outputs (AKA JSON mode).
If you want an LLM to refer to a specific piece of text, give each one an ID and then work with those IDs.
Total anecdote, but I worked on this for a bit for a research-level-code code editor (system paper to come soon, fingers crossed!) and found that basic find-and-replace was pretty brittle. I also had to be confident the source appears only once (not always the case for my use case), and there was a tradeoff of fuzziness of match / likelihood of perfectly correct source.
But yeah, diffs are super hard because the format requires far context and accurate mathematical computation.
Ultimately, the version of this that worked the best for me was a total hack:
Prefix every line of the code with L#### -- the line number. Ask for diffs to be the original text and the complete replacement text including the line number prefix on both original and replacement. Then, to apply, fuzzy match on both line number and context.
I suspect this worked as well as it did because it transmutes the math and computation problems into pattern-matching and copying problems, which LLMs are (still) much better at these days.