Implementing a Simple University Paper Plagiarism Detection System in Python

Building a complete university paper plagiarism detection system involves complex text processing, similarity computation, and database operations, and requires a large corpus of published papers together with appropriate copyright permissions.

Typical systems use algorithms such as cosine similarity, Jaccard similarity, and TF‑IDF weighting, and perform preprocessing steps including tokenization, stop‑word removal, and stemming.

In practice, it is recommended to use professional plagiarism services (e.g., Turnitin, Grammarly, CNKI) that provide mature and compliant solutions; however, if you develop your own system you must also consider data acquisition, user‑interface design, report generation, security, and privacy, possibly leveraging cloud services and big‑data technologies.

The following example demonstrates how to use the NLTK library for preprocessing, compute TF‑IDF weights, calculate cosine similarity, and compare a query paper against a simple in‑memory dictionary representing a paper database.

import re
import string
from collections import Counter
from math import sqrt
from typing import Dict, List, Tuple
import nltk
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize
# 假设有一个简单的论文数据库（实际应用中应使用数据库系统）
SIMPLE_PAPER_DATABASE: Dict[str, str] = {
"paper1": "This is the content of the first paper.",
"paper2": "The second paper has its own unique content.",
# ... 更多论文
}
# 文本预处理函数
def preprocess_text(text: str) -> List[str]:
"""
Perform basic text preprocessing: lowercase, remove punctuation, tokenize, and remove stop words.
Args:
text (str): The input text.
Returns:
list[str]: A list of preprocessed tokens.
"""
text = text.lower().translate(str.maketrans('', '', string.punctuation))
tokens = word_tokenize(text)
stop_words = set(stopwords.words('english'))
preprocessed_tokens = [token for token in tokens if token not in stop_words]
return preprocessed_tokens
# 计算TF-IDF权重
def compute_tfidf(tokens: List[str], all_tokens: List[List[str]]) -> Dict[str, float]:
"""
Compute TF-IDF weights for a given list of tokens, based on a list of all tokens across all documents.
Args:
tokens (list[str]): Tokens from a single document.
all_tokens (list[list[str]]): List of tokens for all documents.
Returns:
dict[str, float]: Dictionary mapping each token to its TF-IDF weight.
"""
token_counts = Counter(tokens)
total_tokens = sum(len(doc_tokens) for doc_tokens in all_tokens)
tfidf_weights = {}
for token, count in token_counts.items():
df = sum(token in doc_tokens for doc_tokens in all_tokens) / len(all_tokens)
idf = log((len(all_tokens) + 1) / (df + 1))  # Smoothed IDF
tfidf_weights[token] = count / len(tokens) * idf
return tfidf_weights
# 计算余弦相似度
def cosine_similarity(vec1: Dict[str, float], vec2: Dict[str, float]) -> float:
"""
Compute the cosine similarity between two dictionaries representing token-weight vectors.
Args:
vec1 (dict[str, float]): Token-weight vector 1.
vec2 (dict[str, float]): Token-weight vector 2.
Returns:
float: Cosine similarity value between 0 and 1, where 1 indicates identical vectors.
"""
dot_product = sum(vec1.get(token, 0) * vec2.get(token, 0) for token in set(vec1.keys()) | set(vec2.keys()))
norm1 = sqrt(sum(val ** 2 for val in vec1.values()))
norm2 = sqrt(sum(val ** 2 for val in vec2.values()))
if norm1 == 0 or norm2 == 0:
return 0.0
return dot_product / (norm1 * norm2)
def compare_papers(paper_id: str, query_paper_content: str) -> Dict[str, float]:
"""
Compare a query paper's content against all papers in the database, returning a dictionary of similarity scores.
Args:
paper_id (str): ID of the query paper.
query_paper_content (str): Content of the query paper.
Returns:
dict[str, float]: Dictionary mapping each paper ID to its similarity score with the query paper.
"""
query_tokens = preprocess_text(query_paper_content)
all_tokens = [preprocess_text(content) for content in SIMPLE_PAPER_DATABASE.values()]
query_tfidf = compute_tfidf(query_tokens, all_tokens)
similarities = {}
for other_paper_id, other_paper_content in SIMPLE_PAPER_DATABASE.items():
if other_paper_id == paper_id:  # Skip comparing the paper with itself
continue
other_tokens = preprocess_text(other_paper_content)
other_tfidf = compute_tfidf(other_tokens, all_tokens)
similarity = cosine_similarity(query_tfidf, other_tfidf)
similarities[other_paper_id] = similarity
return similarities
# 示例用法
query_paper_id = "paper1"
query_paper_content = "This is the content of the first paper, with some added text."
results = compare_papers(query_paper_id, query_paper_content)
for other_paper_id, similarity in results.items():
print(f"Similarity between '{query_paper_id}' and '{other_paper_id}': {similarity * 100:.2f}%")

For real‑world deployment you would need scalable storage (e.g., MySQL, MongoDB), distributed processing frameworks (e.g., Spark) for large datasets, a user‑friendly web or desktop UI, compliance with copyright laws, more sophisticated preprocessing (stemming, lemmatization, synonym replacement), and advanced similarity methods such as sentence‑level metrics or deep‑learning models.