.png){kind=icon}
Return
Gotcha! This Model Uses My Code! Evaluating Membership Leakage Risks in Code Models
.png){kind=icon}
OA
AI
DOI：10.1109/TSE.2024.3482719.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Leveraging large-scale datasets from open-source projects and advances in large language models, recent progress has led to sophisticated code models for key software engineering tasks, such as program repair and code completion. These models are trained on data from various sources, including public open-source projects like GitHub and private, confidential code from companies, raising significant privacy concerns. This paper investigates a crucial but unexplored question: What is the risk of membership information leakage in code models? Membership leakage refers to the vulnerability where an attacker can infer whether a specific data point was part of the training dataset. We present Gotcha, a novel membership inference attack method designed for code models, and evaluate its effectiveness on Java-based datasets. Gotcha simultaneously considers three key factors: model input, model output, and ground truth. Our ablation study confirms that each factor significantly enhances attack performance. Our ablation study confirms that each factor significantly enhances attack performance. Our investigation reveals a troubling finding: membership leakage risk is significantly elevated. While previous methods had accuracy close to random guessing, Gotcha achieves high precision, with a true positive rate of 0.95 and a low false positive rate of 0.10. We also demonstrate that the attacker's knowledge of the victim model (e.g., model architecture and pre-training data) affects attack success. Additionally, modifying decoding strategies can help reduce membership leakage risks. This research highlights the urgent need to better understand the privacy vulnerabilities of code models and develop strong countermeasures against these threats.
Keywords:
Codes
Data models
Training data
Training
Information leakage
Software development management
Data privacy
Privacy
Decoding
Source coding
Membership inference attack
privacy
large language models for code
code completion
Journal
IF:
5.6
Papers:
2.9K
Citations:
1.1W
Organization
View more organizations
Cited Papers
ShareSaveShareSaveATG24 Represses Autophagy and Differentiation and Is Essential for Homeostasy of the Flagellar Pocket in Trypanosoma brucei.png)
PLOS ONE
OAAI
PLOS ONEIF0
OAAIShareSaveShareSave View more