@Cmastication @zleap @ben That's not how MOSS, the most widely-used checker, works.

@shriramk @zleap @ben oh how does it work? Clearly my knowledge is outdated.

@shriramk @zleap @ben looks like MOSS uses fingerprinting which is a computationally efficient way to find white space invariant string matches.

https://yangdanny97.github.io/blog/2019/05/03/MOSS

@Cmastication @shriramk @ben

Sounds interesting, so from this I would guess it would look at a string such as

The cat sat on the mat., or The quick brown fox jumps over the lazy dog. From these generate a hash, is this similar to how say md5sum works, as I could write one of the above in a text file, save and generate a md5sum from that, this would be unqie, if you want back in and changed a lower case letter to upper case, or added a comma, it would change the file, and the md5sum would be different. We could then compare the two checksums to see if they match or don't match.

Or am I completely off track here. Given I am not remotely an expert in this.

@Cmastication @shriramk @ben

I have never understood what white space means.

@zleap @ben @shriramk spaces, tabs, line feeds. Stuff that’s white on a printed page.

@Cmastication @ben @shriramk

Ah thanks for clearing that up.

@Cmastication @zleap @ben But I don't this MOSS would be very effective without the language-sensitive parts. Those do a critical bit of normalization to make the fingerprinting actually "work". And that part is definitely not "matching literal strings".

@shriramk @zleap @ben I clearly don't know much except what I read in the link above, but the language sensitive bits seem to be in a "pre processing" step where certain key words are omitted before removal of whitespace. That makes sense, intuitively.

@Cmastication @zleap @ben They are. My point is that without them, it would be much less useful. So it's not JUST "string matching". That would be pretty easy to defeat.

@shriramk @ben @zleap @Cmastication agreed, the big wins for MOSS are arguably just language-specific tokenization, removing white space and comments, and canonicalizing all variable names. Fingerprinting is easily foiled, including automatically. See our Mossad paper for more - https://arxiv.org/abs/2010.01700

@emeryberger @shriramk @ben @zleap solid work.

@Cmastication @shriramk @zleap @ben I don’t think ChatGPT (yet) knows enough to foil MOSS at scale (as MOSSAD can - taking one source and producing hundreds of variants that all defeat MOSS) but an appropriately tuned prompt would go a long way.

@emeryberger @shriramk @zleap @ben that makes sense to me. I was being a bit cheeky with my example.

@zleap @shriramk @ben @Cmastication There is current ongoing work by Scott Aaronson at OpenAI and other groups (this looks promising: https://arxiv.org/abs/2301.10226) on effective watermarking strategies

@emeryberger @zleap @ben @Cmastication
I've heard about this but I really wonder how useful it can be for, eg, small programs.

@shriramk @zleap @ben @Cmastication There's a very nice blog post about this method, but it was aimed at text, not code, which could easily require different techniques.

https://aiguide.substack.com/p/on-detecting-whether-text-was-generated

@shriramk @Cmastication @zleap @ben
I would be careful calling it the most widely used text-matching software. Thrre are many systems with many variants on how they work.

@WiseWoman @Cmastication @zleap @ben
I am sorry, you are of course absolutely correct. I meant "for software in courses", and forgot the broader context.

Incidentally, Prof. Weber-Wolff, your Nature article on plagiarism detectors [https://www.nature.com/articles/d41586-019-00893-5] is one of the readings in my introductory computer science course, tied to an assignment about document comparison [https://cs.brown.edu/courses/cs019/2022/docdiff.html#%28part._dd-rcs%29]. That article is more relevant today than ever!

@shriramk @Cmastication @zleap @ben
Very cool exercise, I'm honored! But are you sure of the trustworthiness of the similarity checker you are using? They don't even have an imprint, a business address, listed. There is just a form for you to fill out to contact them. And they set a LOT of cookies...

@WiseWoman @Cmastication @zleap @ben No, it's almost certainly a bullshit tool. If you have a suggestion for something better, I'd welcome it.

But I just wanted to give students a bit of a feel without taking too long.

The students do come up with some very creative defenses. I hope that, in turn, they will remember this when they need to do text processing and think *they* can easily implement a notion of similarity (which the programming assignment might have left them thinking).

@shriramk @Cmastication @zleap @ben

If you can afford 3 cents a scan, Copyscape would be usable https://www.copyscape.com/
It is from the people who made Google Alerts. It doesn't store anything, but does cost per shot. Most of the "free" ones are of course in it for getting the text you are testing and data about you.

Oh yes, the "I'll code up a similarity checker this weekend" mindset :)

@WiseWoman @Cmastication @zleap @ben
Thanks for the reco! Will look into it in the summer when I next do course-prep.

We are already paying for GPT-3, now we can pay for a service that also tells us how well GPT-3 is doing I guess. <-;

@Cmastication @ben

Ah, that makes sense,