Reasons for Increased Scrutiny of AI Systems

AI models like ChatGPT and Claude are powerful, but they aren’t perfect. They can sometimes produce inaccurate, biased, or misleading answers due to issues related to data quality, training methods, prompt handling, context management, and system deployment. These problems arise from the complex interaction between model design, user input, and infrastructure. Here are the main factors that explain why incorrect outputs occur: 1. Model Training Limitations AI relies on the data it is trained on. Gaps, outdated information, or insufficient coverage of niche topics lead to shallow reasoning, overfitting to common patterns, and poor handling of rare scenarios. 2. Bias & Hallucination Issues Models can reflect social biases or create “hallucinations,” which are confident but false details. This leads to made-up facts, skewed statistics, or misleading narratives. 3. External Integration & Tooling Issues When AI connects to APIs, tools, or data pipelines, miscommunication, outdated integrations, or parsing errors can result in incorrect outputs or failed workflows. 4. Prompt Engineering Mistakes Ambiguous, vague, or overloaded prompts confuse the model. Without clear, refined instructions, outputs may drift off-task or omit key details. 5. Context Window Constraints AI has a limited memory span. Long inputs can cause it to forget earlier details, compress context poorly, or misinterpret references, resulting in incomplete responses. 6. Lack of Domain Adaptation General-purpose models struggle in specialized fields. Without fine-tuning, they provide generic insights, misuse terminology, or overlook expert-level knowledge. 7. Infrastructure & Deployment Challenges Performance relies on reliable infrastructure. Problems with GPU allocation, latency, scaling, or compliance can lower accuracy and system stability. Wrong outputs don’t mean AI is "broken." They show the challenge of balancing data quality, engineering, context management, and infrastructure. Tackling these issues makes AI systems stronger, more dependable, and ready for businesses. #LLM

I'm increasingly convinced that we need to treat "AI privacy" as a distinct field within privacy, separate from but closely related to "data privacy". Just as the digital age required the evolution of data protection laws, AI introduces new risks that challenge existing frameworks, forcing us to rethink how personal data is ingested and embedded into AI systems. Key issues include: 🔹 Mass-scale ingestion – AI models are often trained on huge datasets scraped from online sources, including publicly available and proprietary information, without individuals' consent. 🔹 Personal data embedding – Unlike traditional databases, AI models compress, encode, and entrench personal data within their training, blurring the lines between the data and the model. 🔹 Data exfiltration & exposure – AI models can inadvertently retain and expose sensitive personal data through overfitting, prompt injection attacks, or adversarial exploits. 🔹 Superinference – AI uncovers hidden patterns and makes powerful predictions about our preferences, behaviours, emotions, and opinions, often revealing insights that we ourselves may not even be aware of. 🔹 AI impersonation – Deepfake and generative AI technologies enable identity fraud, social engineering attacks, and unauthorized use of biometric data. 🔹 Autonomy & control – AI may be used to make or influence critical decisions in domains such as hiring, lending, and healthcare, raising fundamental concerns about autonomy and contestability. 🔹 Bias & fairness – AI can amplify biases present in training data, leading to discriminatory outcomes in areas such as employment, financial services, and law enforcement. To date, privacy discussions have focused on data - how it's collected, used, and stored. But AI challenges this paradigm. Data is no longer static. It is abstracted, transformed, and embedded into models in ways that challenge conventional privacy protections. If "AI privacy" is about more than just the data, should privacy rights extend beyond inputs and outputs to the models themselves? If a model learns from us, should we have rights over it? #AI #AIPrivacy #Dataprivacy #Dataprotection #AIrights #Digitalrights

⚠️Privacy Risks in AI Management: Lessons from Italy’s DeepSeek Ban⚠️ Italy’s recent ban on #DeepSeek over privacy concerns underscores the need for organizations to integrate stronger data protection measures into their AI Management System (#AIMS), AI Impact Assessment (#AIIA), and AI Risk Assessment (#AIRA). Ensuring compliance with #ISO42001, #ISO42005 (DIS), #ISO23894, and #ISO27701 (DIS) guidelines is now more material than ever. 1. Strengthening AI Management Systems (AIMS) with Privacy Controls 🔑Key Considerations: 🔸ISO 42001 Clause 6.1.2 (AI Risk Assessment): Organizations must integrate privacy risk evaluations into their AI management framework. 🔸ISO 42001 Clause 6.1.4 (AI System Impact Assessment): Requires assessing AI system risks, including personal data exposure and third-party data handling. 🔸ISO 27701 Clause 5.2 (Privacy Policy): Calls for explicit privacy commitments in AI policies to ensure alignment with global data protection laws. 🪛Implementation Example: Establish an AI Data Protection Policy that incorporates ISO27701 guidelines and explicitly defines how AI models handle user data. 2. Enhancing AI Impact Assessments (AIIA) to Address Privacy Risks 🔑Key Considerations: 🔸ISO 42005 Clause 4.7 (Sensitive Use & Impact Thresholds): Mandates defining thresholds for AI systems handling personal data. 🔸ISO 42005 Clause 5.8 (Potential AI System Harms & Benefits): Identifies risks of data misuse, profiling, and unauthorized access. 🔸ISO 27701 Clause A.1.2.6 (Privacy Impact Assessment): Requires documenting how AI systems process personally identifiable information (#PII). 🪛 Implementation Example: Conduct a Privacy Impact Assessment (#PIA) during AI system design to evaluate data collection, retention policies, and user consent mechanisms. 3. Integrating AI Risk Assessments (AIRA) to Mitigate Regulatory Exposure 🔑Key Considerations: 🔸ISO 23894 Clause 6.4.2 (Risk Identification): Calls for AI models to identify and mitigate privacy risks tied to automated decision-making. 🔸ISO 23894 Clause 6.4.4 (Risk Evaluation): Evaluates the consequences of noncompliance with regulations like #GDPR. 🔸ISO 27701 Clause A.1.3.7 (Access, Correction, & Erasure): Ensures AI systems respect user rights to modify or delete their data. 🪛 Implementation Example: Establish compliance audits that review AI data handling practices against evolving regulatory standards. ➡️ Final Thoughts: Governance Can’t Wait The DeepSeek ban is a clear warning that privacy safeguards in AIMS, AIIA, and AIRA aren’t optional. They’re essential for regulatory compliance, stakeholder trust, and business resilience. 🔑 Key actions: ◻️Adopt AI privacy and governance frameworks (ISO42001 & 27701). ◻️Conduct AI impact assessments to preempt regulatory concerns (ISO 42005). ◻️Align risk assessments with global privacy laws (ISO23894 & 27701).   Privacy-first AI shouldn't be seen just as a cost of doing business, it’s actually your new competitive advantage.

We’ve reached a point where AI can create “perfect” illusions - right down to convincing identity documents that have no real-world basis. An image circulating recently shows what appears to be an official ID, yet every detail (including the background and text) is entirely fabricated by AI. This isn’t just a hypothetical risk; some people are already mass-producing these fake credentials at an alarming pace online. Why It’s Concerning - Unprecedented Scale: Automation lets fraudsters churn out large volumes of deepfakes quickly, making them harder to detect through manual review alone. - Enhanced Realism: AI systems can generate documents with realistic holograms, security patterns, and microprint, fooling basic validation checks. - Low Entry Barrier: Anyone with a decent GPU and some technical know-how can build - or access - tools for creating synthetic IDs, expanding fraud opportunities beyond sophisticated criminal rings. Preparing for Tomorrow’s Threats Traditional “document checks” used in some countries may not suffice. We need wide spread AI-assisted tools that can spot anomalies in ID documents at scale - such as inconsistent geometry, pixel-level artifacts, or mismatched data sources. Biometrics (e.g., facial recognition, voice authentication) can add layers of identity proof, but these systems also need to be tested against deepfakes. Spoof detection technologies (like liveness checks) can help confirm whether a user’s biometric data is genuine. Probably more than ever it is important for governments to provide smaller businesses means of cross-checking IDs with authoritative databases - whether government, financial, or otherwise. As AI-based fraud techniques evolve, so must our defenses. Keeping pace involves embracing advanced, adaptive technologies for identity verification and maintaining an informed, proactive stance among staff and consumers alike. Do you see biometric verification or real-time data cross-referencing as the most promising approach to identify fake IDs? #innovation #technology #future #management #startups

Have you seen Stanford University's new AI Index Report? There's a ton to digest, but this takeaway stands out to me the most: “The responsible AI ecosystem evolves—unevenly.” In the report, the editors highlight that AI-related incidents are on the rise, but standardized evaluations for validating response quality (and safety) leave MUCH to be desired. From a corporate perspective, there’s an observable gap between understanding the RISKS of errant model responses— and actually taking meaningful ACTION (at the data, system, code, and model response levels) to mitigate it. The primary thrust of the AI movement seems to be this: Build, build, and build some more…then process the consequences later. I think we need to take a step back as data leaders and ask if that’s really an acceptable approach. Should it be? A couple of bright spots highlighted in the report included a few new benchmarks which all offer promising steps toward assessing the factuality and safety of model responses: - HELM Safety - AIR-Bench - and FACTS Governments are also taking notice. “In 2024, global cooperation on AI governance intensified, with organizations including the OECD, EU, U.N., and African Union releasing frameworks focused on transparency, trustworthiness, and other core responsible AI principles.” When it comes to AI, it’s not just our customers who suffer when things go awry. Our stakeholders, our teammates, and our reputations are all on the hook for generative missteps. In short, rewards may still be high—but the risks have never been higher. What do you think? Let me know in the comments!

The Office of the Governor of California published the report "Benefits and Risks of Generative Artificial Intelligence Report" outlining the potential benefits and #risks that generative #AI could bring to the state's government. While this report was a requirement under California's Executive Order N-12-23, the findings could be applied to any other state, government, or organization using #generativeAI. The report starts providing a comparison between conventional #artificialintelligence and generative AI. Then, it lists six major case uses for the #technology: 1. Improve the performance, capacity, and efficiency of ongoing work, research, and analysis through summarization and classification. By analyzing hundreds of millions of data points simultaneously, GenAI can create comprehensive summaries of any collection of artifacts and also categorize and classify information by topic, format, tone, or theme. 2. Facilitate the design of services and products to improve access to people’s diverse needs, across geography and demography. #GenAI can recommend ways to display complex information in a way that resonates best with various audiences or highlight information from multiple sources that is relevant to an individual person. 3. Improve communications in multiple languages and formats to be more accessible to and inclusive of all residents. 4. Improve operations by optimizing software coding and explaining and categorizing unfamiliar code. 5. Find insights and predict key outcomes in complex datasets to empower and support decision-makers. 6. Optimize resource allocation, maximizing energy efficiency and demand flexibility, and promoting environmentally sustainable policies. The report then considers the #risks presented by #generative AI, including: - AI systems could be inaccurate, unreliable, or create misleading or false information. - New GenAI models trained on self-generated, synthetic #data, could negatively impact model performance through training feedback loops. - Input prompts could push the GenAI model to recommend hazardous decisions (#disinformation, #cybersecurity, warfare, promoting violence or racism). - GenAI tools may also be used by bad actors to access information or attack #systems. - As models are increasingly able to learn and apply human psychology, models could be used to create outputs to influence human beliefs, manipulate people's behaviours, or spread #disinformation. - Governance concerns with open-source AI models third-parties that could host models without transparent safety guardrails. - Difficulty in auditing large volumes of training data for the models and tracing the original citation sources for references within the generated content. - Uncertainty over liability for harmful or misleading content generated by the AI. - Complexity and opaqueness of AI model architectures. - The output of GenAI does not reflect social or cultural nuances of subsets of the population.

A new California bill, SB 1047, could introduce restrictions on artificial intelligence, requiring companies to test the safety of AI technologies and making them liable for any serious harm caused by their systems. California is debating SB 1047, a bill that could reshape how AI is developed and regulated. If passed, it would require tech companies to conduct safety tests on powerful AI technologies before release. The bill also allows the state to take legal action if these technologies cause harm, which has sparked concern among major AI companies. Proponents believe the bill will help prevent AI-related disasters, while critics argue it could hinder innovation, particularly for startups and open-source developers. 🛡️ Safety First: SB 1047 mandates AI safety testing before companies release new technologies to prevent potential harm. ⚖️ Legal Consequences: Companies could face lawsuits if their AI systems cause significant damage, adding a new layer of liability. 💻 Tech Industry Pushback: Tech giants like Google, Meta, and OpenAI are concerned that the bill could slow AI innovation and create legal uncertainties. 🔓 Impact on Open Source: The bill might limit open-source AI development, making it harder for smaller companies to compete with tech giants. 🌐 Potential Global Effects: If passed, the bill could set a precedent for AI regulations in other states and countries, influencing the future of AI governance globally. #AI #AIBill #TechRegulation #CaliforniaLaw #ArtificialIntelligence #OpenSource #Innovation #TechPolicy #SB1047 #AIRegulation 

A new behavior that must be evaluated in AI models: sycophancy. (And don’t worry if you had to look up what that means—I did too.) On April 25th, OpenAI released a new version of GPT-4o in ChatGPT. But something was off. The model had become noticeably more agreeable—to the point of being unhelpful or even harmful. It wasn’t just being nice; it was validating doubts, encouraging impulsive behavior, and reinforcing negative emotions. The cause? New training signals like thumbs-up/down user feedback unintentionally weakened safeguards against sycophantic behavior. And since sycophancy hadn’t been explicitly tracked or flagged in previous evaluations, it slipped through. What I appreciated most was OpenAI’s transparency in owning the miss and outlining clear steps for improvement. It's a powerful reminder that as we release more advanced AI systems, new risks will emerge—ones we may not yet be measuring. I believe this signals a rising need for AI quality control—what I like to call QA for AI, or even “therapists for AI.” People whose job is to question, test, and ensure the model is sane, safe, and aligned before it reaches the world. We’re still learning and evolving with these tools—and this post is a great read if you're following the path of responsible AI: https://lnkd.in/gXwY-Rjf

Should AI be regulated? Some thoughts about President Biden's recent Executive Order (Part 1)  There are 1000+ companies in Techstars portfolio that integrate AI/ML in their business. Which means that the question of AI regulation is very much top of mind for us. President Biden's recent Executive Order is the first comprehensive attempt by the U.S. government to do so. As someone generally in favor of a regulatory framework that sets clear boundaries & guidelines for businesses, this initiative is a commendable first step. It highlights crucial areas like safety, equity, and responsible AI use, acknowledging both the technology's potential and its risks. But… The EO also has notable shortcomings, particularly regarding its broad definition of AI systems, and potential conflicts/overlaps with existing regulations. 1) Broad Definition of AI Systems: The order defines AI systems broadly, including “any data system, software, hardware, application, tool, or utility that operates in whole or in part using AI". This expansive definition could encompass a wide range of technologies, from highly advanced AI-driven systems to more basic software that incorporates AI elements to a minimal degree (like a simple recommendation algorithm or a basic data analysis tool).This could lead to overregulation and unnecessary burdens for a wide range of developers and products. 2) Potential Conflicts with Existing Regulations: - Privacy Laws: The federal AI regulations might conflict with established privacy laws like GDPR or CCPA, especially in terms of data-handling and consumer rights, leading to complex compliance scenarios for international companies - Industry-Specific Challenges: E.g. in healthcare, AI regulations could complicate the FDA’s established process for medical devices, especially for AI-based diagnostic tools - Federal vs. State Law Conflicts: States with their own technology laws, could face conflicts with Federal AI regulations. California's CCPA is a prime example, where state-specific data privacy rules could intersect with federal AI regulations. Additionally, federal regulations might preempt more stringent state-level protections, leading to legal challenges and debates over federal versus state autonomy. 3) Feasibility concerns:  The rapid evolution and complexity of AI will without a doubt outpace the capabilities of current regulatory frameworks, and the EO is attempting to cover too many bases at once in a fairly rigid and exhaustive way that can’t work for a technology that is evolving so fast. For instance, the order proposes using AI watermarking for content verification, but this technology is still in development and may not yet be reliable enough for widespread implementation. Regulations based on emerging or incomplete technologies can be ineffective or even counterproductive. In my follow-up post, I’ll explore how the answer lies in developing flexible adaptive regulations that can evolve with the technology. 

No matter where I am—earlier this week it was England, today it’s Germany, and later this month it’s India—the conversation is the same. CISOs everywhere are asking: How do we secure AI? It’s not theoretical anymore. Models are live. Risks are active. And attackers are already finding ways in. Here’s what we’re seeing: ➡️ 80+% of CISOs own AI security and safety risk for their organizations ➡️ AI systems are being jailbroken, manipulated, and misused in production ➡️ These systems introduce entirely new security challenges that require purpose-built defenses ➡️ Those purpose-built defenses require a combination of human insight and automation; a hybrid testing approach is essential ➡️ Leaders like Anthropic and Snap Inc. are engaging ethical hackers to pressure-test their AI models and systems If we treat AI like any other system, we’ll miss what makes it uniquely vulnerable. Securing it means rethinking assumptions—and widening the circle of trust. More on how forward-looking CISOs are responding (link in comments). #CISO #AI #Cybersecurity #SecurityResearchers #OffensiveSecurity #ReturnOnMitigation #TrustByDesign #HumanIngenuity #Leadership #WeAreDevelopers