Data Privacy Risks When Using AI Tools

This new white paper by Stanford Institute for Human-Centered Artificial Intelligence (HAI) titled "Rethinking Privacy in the AI Era" addresses the intersection of data privacy and AI development, highlighting the challenges and proposing solutions for mitigating privacy risks. It outlines the current data protection landscape, including the Fair Information Practice Principles, GDPR, and U.S. state privacy laws, and discusses the distinction and regulatory implications between predictive and generative AI. The paper argues that AI's reliance on extensive data collection presents unique privacy risks at both individual and societal levels, noting that existing laws are inadequate for the emerging challenges posed by AI systems, because they don't fully tackle the shortcomings of the Fair Information Practice Principles (FIPs) framework or concentrate adequately on the comprehensive data governance measures necessary for regulating data used in AI development. According to the paper, FIPs are outdated and not well-suited for modern data and AI complexities, because: - They do not address the power imbalance between data collectors and individuals. - FIPs fail to enforce data minimization and purpose limitation effectively. - The framework places too much responsibility on individuals for privacy management. - Allows for data collection by default, putting the onus on individuals to opt out. - Focuses on procedural rather than substantive protections. - Struggles with the concepts of consent and legitimate interest, complicating privacy management. It emphasizes the need for new regulatory approaches that go beyond current privacy legislation to effectively manage the risks associated with AI-driven data acquisition and processing. The paper suggests three key strategies to mitigate the privacy harms of AI: 1.) Denormalize Data Collection by Default: Shift from opt-out to opt-in data collection models to facilitate true data minimization. This approach emphasizes "privacy by default" and the need for technical standards and infrastructure that enable meaningful consent mechanisms. 2.) Focus on the AI Data Supply Chain: Enhance privacy and data protection by ensuring dataset transparency and accountability throughout the entire lifecycle of data. This includes a call for regulatory frameworks that address data privacy comprehensively across the data supply chain. 3.) Flip the Script on Personal Data Management: Encourage the development of new governance mechanisms and technical infrastructures, such as data intermediaries and data permissioning systems, to automate and support the exercise of individual data rights and preferences. This strategy aims to empower individuals by facilitating easier management and control of their personal data in the context of AI. by Dr. Jennifer King Caroline Meinhardt Link: https://lnkd.in/dniktn3V

China's DeepSneak AI: This is your warning. It's faster, and cheaper, but you need to know why. 🚨 Chinese DeepSeek AI may be turning heads with its impressive performance, companies prioritizing data privacy should think twice before jumping on board. Here's why: ## The Privacy Pitfall DeepSeek's privacy policy raises serious red flags: - All data stored in China 🇨🇳 - Indefinite data retention - Broad usage rights for collected information This combination creates a perfect storm of privacy concerns for businesses handling sensitive data. ## The Hidden Costs Sure, DeepSeek might seem like a bargain compared to other AI solutions. But remember: - Your data is the real currency - Long-term privacy risks outweigh short-term savings - Potential for data misuse or unauthorized access ## Why It Matters For companies, using DeepSeek could lead to: - Breach of client confidentiality - Exposure of trade secrets - Compliance violations with data protection laws - Reputational damage if a data leak occurs ## The Bigger Picture This situation highlights a broader trend: the need for careful vetting of AI tools, especially those originating from countries with different data protection standards. Don't let the allure of cutting-edge AI cloud your judgment. When it comes to protecting your company's information, privacy should always be the top priority. Remember: If a deal seems too good to be true, it probably is – especially in the world of AI and data privacy.

The Cybersecurity and Infrastructure Security Agency together with the National Security Agency, the Federal Bureau of Investigation (FBI), the National Cyber Security Centre, and other international organizations, published this advisory providing recommendations for organizations in how to protect the integrity, confidentiality, and availability of the data used to train and operate #artificialintelligence. The advisory focuses on three main risk areas: 1. Data #supplychain threats: Including compromised third-party data, poisoning of datasets, and lack of provenance verification. 2. Maliciously modified data: Covering adversarial #machinelearning, statistical bias, metadata manipulation, and unauthorized duplication. 3. Data drift: The gradual degradation of model performance due to changes in real-world data inputs over time. The best practices recommended include: - Tracking data provenance and applying cryptographic controls such as digital signatures and secure hashes. - Encrypting data at rest, in transit, and during processing—especially sensitive or mission-critical information. - Implementing strict access controls and classification protocols based on data sensitivity. - Applying privacy-preserving techniques such as data masking, differential #privacy, and federated learning. - Regularly auditing datasets and metadata, conducting anomaly detection, and mitigating statistical bias. - Securely deleting obsolete data and continuously assessing #datasecurity risks. This is a helpful roadmap for any organization deploying #AI, especially those working with limited internal resources or relying on third-party data.

🧠 “Data systems are designed to remember data, not to forget data.” – Debbie Reynolds, The Data Diva 🚨 I just published a new essay in the Data Privacy Advantage newsletter called: 🧬An AI Data Privacy Cautionary Tale: Court-Ordered Data Retention Meets Privacy🧬 🧠 This essay explores the recent court order from the United States District Court for the Southern District of New York in the New York Times v. OpenAI case. The court ordered OpenAI to preserve all user interactions, including chat logs, prompts, API traffic, and generated outputs, with no deletion allowed, not even at the user's request. 💥 That means: 💥“Delete” no longer means delete 💥API business users are not exempt 💥Personal, confidential, or proprietary data entered into ChatGPT could now be locked in indefinitely 💥Even if you never knew your data would be involved in litigation, it may now be preserved beyond your control 🏛️ This order overrides global privacy laws, such as the GDPR and CCPA, highlighting how litigation can erode deletion rights and intensify the risks associated with using generative AI tools. 🔍 In the essay, I cover: ✅ What the court order says and why it matters ✅ Why enterprise API users are directly affected ✅ How AI models retain data behind the scenes ✅ The conflict between privacy laws and legal hold obligations ✅ What businesses should do now to avoid exposure 💡 My recommendations include: • Train employees on what not to submit to AI • Curate all data inputs with legal oversight • Review vendor contracts for retention language • Establish internal policies for AI usage and audits • Require transparency from AI providers 🏢 If your organization is using generative AI, even in limited ways, now is the time to assess your data discipline. AI inputs are no longer just temporary interactions; they are potentially discoverable records. And now, courts are treating them that way. 📖 Read the full essay to understand why AI data privacy cannot be an afterthought. #Privacy #Cybersecurity #datadiva#DataPrivacy #AI #LegalRisk #LitigationHold #PrivacyByDesign #TheDataDiva #OpenAI #ChatGPT #Governance #Compliance #NYTvOpenAI #GenerativeAI #DataGovernance #PrivacyMatters

PSA: If you're using any "free" AI tools, you should assume your data (i.e. whatever you put into the tool and how you interact with it) is your payment for gaining access. AI is incredibly expensive to run. It demands massive amounts of money, energy, and very pricey (and hard to get) Nvidia chips. The immense computing power behind AI isn't just being offered out of generosity - there's almost always going to be some type of a business motive. There's no inherent right to free AI access, so if a company offers free tools, they're likely to be monetizing or benefiting from your data in some way - often by using it to train their models. For marketing leaders, this is absolutely critical to understand. If your teams use any free AI tools, or you're letting them figure out their tool kit completely on their own without checking on this, or you are not providing powerful company-approved data-safe AI tools, you need to change course. (I talk to a lot of leaders who do this, and don't understand they are operationalizing a major internal data risk) It's your responsibility to scrutinize the data training policies of every tool they use. (And even if you say "no AI" as a policy at your company, I PROMISE YOU, at least some people on your team are using their own free tools without you even knowing it!) Evaluate everything, even tools from well-known companies. A familiar brand name doesn't necessarily guarantee data safety in a free or "personal" account. In fact, the bigger the name, the more cautious you may want to be. ⚠️👉Always assume that, unless you check data training policies on the exact account type you're using and are proven otherwise, free (and some paid) AI tools are most-likely training on your data or using it for their own benefit in some other capacity. Some platforms do let you opt out of data training, but you must adjust the settings yourself. In ChatGPT's Free version, you can turn off data training. You need to do it manually in the ChatGPT Plus and Pro paid accounts, too. (Teams and Enterprise accounts made for businesses have data training set to off and you can't turn it back on) To turn data training off in free and paid personal tier ChatGPT accounts, go to "Settings" > "Data Controls" > and toggle "improve the model for everyone" to OFF. Similarly, Google Gemini's data policies vary across free, personal paid, and business accounts. Just because your business Workspace account doesn't train on data and your company approved that, it doesn't mean the free or personal paid accounts follow the exact same rules. Don't assume it's safe just because your company approved it. Always check. Here's your reminder for a lot of life, and especially for AI  👉  If you're not paying for a valuable product with money, YOU are probably the product. Be intentional and vigilant about where you put your data. Or at least make an informed decision that you're okay with whatever they do with whatever you give it.

The EDPB recently published a report on AI Privacy Risks and Mitigations in LLMs.   This is one of the most practical and detailed resources I've seen from the EDPB, with extensive guidance for developers and deployers. The report walks through privacy risks associated with LLMs across the AI lifecycle, from data collection and training to deployment and retirement, and offers practical tips for identifying, measuring, and mitigating risks.   Here's a quick summary of some of the key mitigations mentioned in the report:   For providers: • Fine-tune LLMs on curated, high-quality datasets and limit the scope of model outputs to relevant and up-to-date information. • Use robust anonymisation techniques and automated tools to detect and remove personal data from training data. • Apply input filters and user warnings during deployment to discourage users from entering personal data, as well as automated detection methods to flag or anonymise sensitive input data before it is processed. • Clearly inform users about how their data will be processed through privacy policies, instructions, warning or disclaimers in the user interface. • Encrypt user inputs and outputs during transmission and storage to protect data from unauthorized access. • Protect against prompt injection and jailbreaking by validating inputs, monitoring LLMs for abnormal input behaviour, and limiting the amount of text a user can input. • Apply content filtering and human review processes to flag sensitive or inappropriate outputs. • Limit data logging and provide configurable options to deployers regarding log retention. • Offer easy-to-use opt-in/opt-out options for users whose feedback data might be used for retraining.   For deployers: • Enforce strong authentication to restrict access to the input interface and protect session data. • Mitigate adversarial attacks by adding a layer for input sanitization and filtering, monitoring and logging user queries to detect unusual patterns. • Work with providers to ensure they do not retain or misuse sensitive input data. • Guide users to avoid sharing unnecessary personal data through clear instructions, training and warnings. • Educate employees and end users on proper usage, including the appropriate use of outputs and phishing techniques that could trick individuals into revealing sensitive information. • Ensure employees and end users avoid overreliance on LLMs for critical or high-stakes decisions without verification, and ensure outputs are reviewed by humans before implementation or dissemination. • Securely store outputs and restrict access to authorised personnel and systems.   This is a rare example where the EDPB strikes a good balance between practical safeguards and legal expectations. Link to the report included in the comments.   #AIprivacy #LLMs #dataprotection #AIgovernance #EDPB #privacybydesign #GDPR

🚨 Using DeepSeek Poses Serious Risks to Your Privacy and Security 🚨 DeepSeek, the AI chatbot developed by a Chinese firm, has gained immense popularity recently. However, beneath its advanced capabilities lie critical security flaws, privacy risks, and potential ties to the Chinese government that make it unsafe for use. Here’s why you should think twice before using DeepSeek: 1. Major Data Breaches and Security Vulnerabilities Exposed Database: DeepSeek recently left over 1 million sensitive records, including chat logs and API keys, openly accessible due to an unsecured database. This exposed user data to potential cyberattacks and espionage. Unencrypted Data Transmission: The DeepSeek iOS app transmits sensitive user and device data without encryption, making it vulnerable to interception by malicious actors. Hardcoded Encryption Keys: Weak encryption practices, such as the use of outdated algorithms and hardcoded keys, further compromise user data security. 2. Ties to the Chinese Government Data Storage in China: DeepSeek stores user data on servers governed by Chinese law, which mandates companies to cooperate with state intelligence agencies. Hidden Code for Data Transmission: Researchers uncovered hidden programming in DeepSeek's code that can transmit user data directly to China Mobile, a state-owned telecommunications company with known ties to the Chinese government. National Security Concerns: U.S. lawmakers and cybersecurity experts have flagged DeepSeek as a tool for potential surveillance, urging bans on its use in government devices. 3. Privacy and Ethical Concerns Extensive Data Collection: DeepSeek collects detailed user information, including chat histories, device data, keystroke patterns, and even activity from other apps. This raises serious concerns about profiling and surveillance. Propaganda Risks: Investigations reveal that DeepSeek's outputs often align with Chinese government narratives, spreading misinformation and censorship on sensitive topics like Taiwan or human rights issues. 4. Dangerous Outputs and Misuse Potential Harmful Content Generation: Studies show that DeepSeek is significantly more likely than competitors to generate harmful or biased content, including extremist material and insecure code. Manipulation Risks: Its vulnerabilities make it easier for bad actors to exploit the platform for phishing scams, disinformation campaigns, and even cyberattacks. What Should You Do? Avoid using DeepSeek for any sensitive or personal information. Advocate for transparency and stricter regulations on AI tools that pose security risks. Stay informed about safer alternatives developed by companies with robust privacy protections. Your data is valuable—don’t let it fall into the wrong hands. Let’s prioritize safety and accountability in AI! 💡

DeepSeek AI: What You Need to Know About Privacy and Security Risks DeepSeek, a powerful AI model from China, is emerging as a strong competitor to U.S.-based AI systems. However, its data collection practices raise serious privacy and cybersecurity concerns. Unlike some open-source AI models that prioritize user privacy, DeepSeek gathers a large amount of user information, often without clear transparency. How DeepSeek Collects Your Data. DeepSeek gathers data in three main ways: 1) What You Enter – This includes any text, audio, prompts, uploaded files, feedback, and chat history you provide when using the platform. 2) Automatically Tracked Data – The platform collects details about your device, such as IP address, operating system, keystroke patterns, cookies, and how you interact with the AI. 3) Third-Party Data – DeepSeek also gathers additional information from external sources, such as advertisers and partners, which may add to the profile they build on you. DeepSeek's technological advancements are impressive, but its data collection and tracking policies warrant scrutiny. As AI adoption accelerates, users should weigh convenience against security risks and remain vigilant about how their data is being used in an increasingly connected world. #AI #Cybersecurity #Privacy #DeepSeek #LLM #DataSecurity