Validation of automated (dis)similarity in legal texts: the case of regional trade agreements

Regulatory patterns within the ever-growing set of regional trade agreements (RTAs) emerge from a continuous cycle of inspiration, paraphrasing, or copy-pasting of provisions from previous treaties. The sheer number and length of RTAs makes it difficult to trace the origin of a given provision, and this is where automated text mining comes to the rescue. In this article we validate the assumptions and then develop automatic tools which make it possible to determine who creates the rules and who copies them. To this end, we split the articles into various units, from short sequences of characters to long sequences of words, and check the similarity of articles using multiple methods of assessing similarity, from simple Jaccard similarity to document embeddings. The results of automatic similarity detection are then compared to a number of expert-created challenges, to discover which method matches the ground truth best. Even though dimensionality reduction outperforms simpler bag-of-words methods in assessing pairwise similarity between articles, it is often mistaken in finding the ultimate source for the article. Jaccard similarity with a low threshold is sufficient to solve the tricky cases. Sequences of words – which increase the importance of minor changes and modifications – fail to improve the results: analysis of single words or sequences of characters is more effective. Even under conservative assumptions, about half of articles in RTAs are copied from another RTA. Our research lends empirical corroboration to quantitative assessment of the rule-making power of actors within the international economic law.