API Integration Challenges

Satya Nadella described a future at Microsoft where there may be more than 20 million agents working alongside employees. This brings up interesting questions about how to monitor what these agents are doing, what these agents need to look like, and what they're allowed to access. Satya believes we need to start with the non-negotiables. Agents need to be fully inspectable and fully auditable. And the moment an agent can write code and execute it, that code has to run in an environment governed by policy. This is one of the engineering challenges of the AI moment; the infra that has to get built as companies stand up the platform for agentic work. If AI is going to amplify human capability at scale, we have to know what our agents are doing, constrain what they can access, and be able to audit and intervene when something goes wrong. It's what makes large-scale deployment possible. It's what earns trust. The alternative is launching millions of autonomous systems into production and hoping for the best.

Deloitte’s new integrated EMEA structure sounds big. On paper, it is exactly what the market demands. Clients are cross border. Capability needs to move seamlessly. Depth of expertise should not be trapped within national boundaries. But we have seen this play out before. The constraint is rarely strategy, it is execution. These firms are still fundamentally built on country P&Ls and local governance models, culturally shaped by Partner politics and deeply embedded autonomy. When a cross-border engagement emerges, the first wave of energy is often spent negotiating internally: who owns the client, how revenue is allocated, who controls the A team, who carries the margin. Too much oxygen is consumed inside the firm before it ever reaches the client. Layer on bureaucracy and it becomes heavier still. Regional structures sit on top of national structures rather than replacing them. Committees multiply. Alignment calls expand. Decision rights blur rather than clarify. Instead of moving faster as a unified region, firms often move slower because every major engagement becomes a negotiation between power centers. Even basic coordination is complicated by the fact that many member firms still operate on different systems and platforms, reinforcing fragmentation. There is also a cultural reality that is harder to confront. Partnership models reward local performance and local loyalty. Integration demands that Partners think beyond their own territory and accept greater transparency around margins and performance. That is uncomfortable. Poorly aligned and integrated systems further compound the difficulty. So the question is not whether integration is the right headline. It is. The real question is whether these firms are able to rewire economics, simplify governance and genuinely prioritize regional value creation over local protection. Without that, it risks becoming another well intentioned restructure that looks integrated externally while remaining federated internally. The only true structural solution would be to move away from the traditional Partnership governance model altogether, because many of the inefficiencies sit at its core. Separation from Audit is a must if they truly want to fulfil their potential.

The Voice Stack is improving rapidly. Systems that interact with users via speaking and listening will drive many new applications. Over the past year, I’ve been working closely with DeepLearning.AI, AI Fund, and several collaborators on voice-based applications, and I will share best practices I’ve learned in this and future posts. Foundation models that are trained to directly input, and often also directly generate, audio have contributed to this growth, but they are only part of the story. OpenAI’s RealTime API makes it easy for developers to write prompts to develop systems that deliver voice-in, voice-out experiences. This is great for building quick-and-dirty prototypes, and it also works well for low-stakes conversations where making an occasional mistake is okay. I encourage you to try it! However, compared to text-based generation, it is still hard to control the output of voice-in voice-out models. In contrast to directly generating audio, when we use an LLM to generate text, we have many tools for building guardrails, and we can double-check the output before showing it to users. We can also use sophisticated agentic reasoning workflows to compute high-quality outputs. Before a customer-service agent shows a user the message, “Sure, I’m happy to issue a refund,” we can make sure that (i) issuing the refund is consistent with our business policy and (ii) we will call the API to issue the refund (and not just promise a refund without issuing it). In contrast, the tools to prevent a voice-in, voice-out model from making such mistakes are much less mature. In my experience, the reasoning capability of voice models also seems inferior to text-based models, and they give less sophisticated answers. (Perhaps this is because voice responses have to be more brief, leaving less room for chain-of-thought reasoning to get to a more thoughtful answer.) When building applications where I need a more control over the output, I use agentic workflows to reason at length about the user’s input. In voice applications, this means I end up using a pipeline that includes speech-to-text (STT) to transcribe the user’s words, then processes the text using one or more LLM calls, and finally returns an audio response to the user via TTS (text-to-speech). This, where the reasoning is done in text, allows for more accurate responses. However, this process introduces latency, and users of voice applications are very sensitive to latency. When DeepLearning.AI worked with RealAvatar (an AI Fund portfolio company led by Jeff Daniel) to build an avatar of me, we found that getting TTS to generate a voice that sounded like me was not very hard, but getting it to respond to questions using words similar to those I would choose was. Even after much tuning, it remains a work in progress. You can play with it at https://lnkd.in/gcZ66yGM [At length limit. Full text, including latency reduction technique: https://lnkd.in/gjzjiVwx ]

We surveyed real estate operators and proptech companies on how open their property management platforms actually are.   No platform scored above a 6.5 out of 10.   The lowest? Just over 3.   The comments were brutal. "Data hostage." "Frankenstein collection of acquired parts." "I'd give zero if I could."   Here's what stood out:   Every major PMS talks about openness. Yardi released an MCP connector for Claude. AppFolio launched a marketplace. Entrata built its brand on open APIs.   But these platforms still have a long way to go in the eyes of the people who use them day-to-day.   Entrata scored highest at 6.2 — but respondents say PE ownership has eroded the open ethos. API access that was free now costs money. Payments APIs still don't exist.   Yardi came in at 5.9. The good news: they're trending toward openness. The bad news: their APIs are still transitioning from SOAP to REST. One respondent put it perfectly — "the 2000s called."   RealPage landed at 3.1. Multiple respondents described the platform as actively hostile to third-party integration. Several said the company holds customer data hostage and refuses to work with perceived competitors.   And this matters more than ever. AI is making platform openness a far higher priority. If your PMS can't connect to third-party tools, your AI strategy is dead on arrival.   The full breakdown — platform by platform, in respondents' own words — is live on Thesis Driven. Link in comments.   Which PMS are you on, and does this match your experience?

APIs have long served as the glue between services in traditional software systems. But as we shift to large-scale AI-driven architectures—especially those powered by Large Language Models (LLMs)—that glue starts to crack. APIs were designed for transactional data exchange, not for the emergent behaviors, stateful interactions, or reasoning loops that LLMs require. The result? Tangled integrations, scattered orchestration, and brittle pipelines. 𝗠𝗖𝗣 (𝗠𝗼𝗱𝗲𝗹 𝗖𝗼𝗻𝘁𝗲𝘅𝘁 𝗣𝗿𝗼𝘁𝗼𝗰𝗼𝗹) steps in with a purpose-built foundation for agentic workflows—where LLMs don’t just call endpoints but collaborate, reason, and adapt. Think: → Clean intent passing → Structured memory → Multi-agent interaction → Contextual decision-making As we build more autonomous and intelligent systems, this transition isn’t optional—it’s inevitable. If you’re still architecting with legacy assumptions, it’s time to rethink your foundation. Agentic AI demands new infrastructure. MCP is that infrastructure. What’s your take—are you seeing the limits of APIs in your LLM projects? These amazing graphic is created by Aakriti Aggarwal !

🚨 BREAKING: OpenAI has just launched ChatGPT Agent. Below are important privacy & security risks everybody should be aware of: Agentic AI applications differ from non-agentic ones particularly regarding the access rights/permissions they require in order to engage with external tools on the user's behalf. The more autonomous the agent, the more permissions/access rights it will require. For example, if a user wants an AI agent to search and buy a dress for them without asking further questions, besides accessing the internet, the AI agent will need to access their wallet. If the user wants the agent to schedule an event and invite friends, it will need access to at least the calendar and contact list. Having said that, any permission given to a third-party app or system has potential privacy and security risks. ChatGPT already presents privacy risks due to the way it's trained, the way it processes personal data, the user's privacy settings, and the type of personal information being input by users. The privacy risks from ChatGPT Agent will be exponentially higher as many people will be giving access rights to external tools containing personal information (calendar, email, wallet, and more). OpenAI knows that malicious actors will try to trick other people's AI agents into sharing private information, including address, email, phone, credit card information, and more. Sam Altman has just posted on X, recommending that people give agents "the minimum access required to complete a task." In many cases, the privacy and security risks of letting an AI agent perform a task will greatly outweigh any productivity benefits it can offer (but people will use AI agents anyway, because of hype, curiosity, or because their company is "AI first") Unfortunately, the pace of AI development is much faster than the pace of AI literacy. Most people haven't yet understood ChatGPT's privacy risks, but they will be thrown a new feature with exponentially MORE risks. - 👉 Never miss my analyses on AI: join my newsletter's 68,200+ subscribers (below).

Voice AI is more than just plugging in an LLM. It's an orchestration challenge involving complex AI coordination across STT, TTS and LLMs, low-latency processing, and context & integration with external systems and tools. Let's start with the basics: ---- Real-time Transcription (STT) Low-latency transcription (<200ms) from providers like Deepgram ensures real-time responsiveness. ---- Voice Activity Detection (VAD) Essential for handling human interruptions smoothly, with tools such as WebRTC VAD or LiveKit Turn Detection ---- Language Model Integration (LLM) Select your reasoning engine carefully—GPT-4 for reliability, Claude for nuanced conversations, or Llama 3 for flexibility and open-source options. ---- Real-Time Text-to-Speech (TTS) Natural-sounding speech from providers like Eleven Labs, Cartesia or Play.ht enhances user experience. ---- Contextual Noise Filtering Implement custom noise-cancellation models to effectively isolate speech from real-world background noise (TV, traffic, family chatter). ---- Infrastructure & Scalability Deploy on infrastructure designed for low-latency, real-time scaling (WebSockets, Kubernetes, cloud infrastructure from AWS/Azure/GCP). ---- Observability & Iterative Improvement Continuous improvement through monitoring tools like Prometheus, Grafana, and OpenTelemetry ensures stable and reliable voice agents. 📍You can assemble this stack yourself or streamline the entire process using integrated API-first platforms like Vapi. Check it out here ➡️https://bit.ly/4bOgYLh What do you think? How will voice AI tech stacks evolve from here?

AI security/securing the use of AI is going to kill me. I use Claude Code almost daily. It's a problem.... Here's what I have to change AGAIN this week. Security researcher Ari Marzuk disclosed 30+ vulnerabilities across AI coding tools. Cursor. GitHub Copilot. Windsurf. Claude Code. All of them. He called it IDEsaster. The attack chain includes prompt injection, hijacking LLM context, and auto-approved tool calls executing without permission. Then, legitimate IDE features are weaponized for data exfiltration and RCE. Your .env files. Your API keys. Your source code. Accessible through features you thought were safe. Most studies I read claim that around 85% of developers now use AI coding tools daily. Most have no idea their IDE treats its own features as inherently trusted. 𝗦𝗼... 𝗮𝗳𝘁𝗲𝗿 𝗿𝗲𝘃𝗶𝗲𝘄𝗶𝗻𝗴 𝗔𝗿𝗶'𝘀 𝗿𝗲𝘀𝗲𝗮𝗿𝗰𝗵, 𝗵𝗲𝗿𝗲'𝘀 𝗜 𝘄𝗶𝗹𝗹 𝗯𝗲 𝗱𝗼𝗶𝗻𝗴... Be warned: All this is SO much easier said than done! Audit every MCP server connection. Checked for tool poisoning vectors where legitimate tools might parse attacker-controlled input from GitHub PRs or web content. Removed servers I couldn't verify. Disabled auto-approve for file writes. The attack chains weaponize configuration files and project instructions like .claude/settings.json and CLAUDE.md. One malicious write to these files can alter agent behavior or achieve code execution without additional user interaction. Move all credentials to a secrets manager. No .gitignored .env files in agent-accessible directories. API keys live in 1Password CLI. Environment variables inject at runtime through a wrapper script the LLM never sees. Start running Claude Code in isolated containers. Mounted volumes limited to specific project directories. No access to ~/.ssh, ~/.aws, or ~/.config. If the agent gets compromised, blast radius stays contained. Enable all security warnings. Claude Code added explicit warnings for JSON schema exfiltration and settings file modifications. These exist because Anthropic knows the attack surface. Add pre-commit hooks for hidden characters. Prompt injections hide in pasted URLs, READMEs, and file names using invisible Unicode. Flag non-ASCII characters in any file the agent might ingest. The fix isn't to stop using AI coding tools. The fix is to stop trusting them implicitly. What controls do you have for AI tools with write access to your codebase? 👉 Follow for more AI and cybersecurity insights with the occasional rant #AISecurity #DevSecOps

Agentic AI can get expensive fast. Not because the model is always wrong. Because every agent step can add more context, more tool calls, more reasoning, and more tokens. If you want scalable agents, token efficiency becomes architecture. Here are 6 ways to save tokens in agentic workflows: → 𝗣𝗿𝗼𝗺𝗽𝘁 𝗖𝗮𝗰𝗵𝗶𝗻𝗴 Reuse repeated system prompts, instructions, and context instead of processing the same input again. Best for stable instructions, long policies, and repeated workflows. → 𝗦𝗲𝗺𝗮𝗻𝘁𝗶𝗰 𝗖𝗮𝗰𝗵𝗶𝗻𝗴 Reuse previous answers when a new query has the same meaning, even if the wording is different. Best for support, FAQs, internal knowledge, and repeated user requests. → 𝗥𝗼𝘂𝘁𝗶𝗻𝗴 Send each task to the right model, tool, or workflow based on complexity, cost, and accuracy needs. Simple tasks should not use your most expensive model. → 𝗞𝗲𝗲𝗽𝗶𝗻𝗴 𝗖𝗼𝗻𝘁𝗲𝘅𝘁 𝗖𝗹𝗲𝗮𝗻 Remove irrelevant, outdated, duplicated, or noisy information before sending context to the model. Clean context improves both cost and output quality. → 𝗖𝗼𝗻𝘁𝗲𝘅𝘁 𝗖𝗼𝗺𝗽𝗮𝗰𝘁𝗶𝗼𝗻 Compress long conversations or documents into shorter summaries while preserving critical facts. Useful when agents need memory without carrying the entire history. → 𝗟𝗮𝘇𝘆-𝗟𝗼𝗮𝗱𝗶𝗻𝗴 𝗖𝗼𝗻𝘁𝗲𝘅𝘁 Load only the information needed at the moment instead of dumping everything upfront. The lesson is simple: More context is not always better. Better context is better. Token savings come from smarter caching, cleaner context, better routing, and loading only what the agent actually needs. Save this if you are building AI agents, RAG systems, automation workflows, or production GenAI apps.

"Maximizing the value of your Claude Code sessions" Agentic coding tools like Claude Code are incredible, but are you getting the most value out of your tokens? Anthropic's Lydia Hallie just dropped a fantastic guide on how to run efficient Claude Code sessions: https://lnkd.in/gujXdtBJ It turns out that fixing the exact same bug can cost completely different amounts depending on your session hygiene. Head's up: I do anticipate Claude will do more of this for you at some point, but until then: Here is how to optimize your workflow and stop wasting context: 1. 🧹 Run /clear between tasks: Don't drag old context into a new problem. You're paying to re-read it on every single turn! 2. ⚙️ Set your /model and /effort upfront: Changing these mid-conversation busts your prompt cache, forcing a full-price prefill of your entire session. 3. 📎 @-mention files directly: Instead of typing out file paths, tagging them attaches the file to your message immediately, saving Claude a roundtrip "Read" call. 4. 🤫 Keep commands quiet: Add quiet flags to noisy terminal commands (like test runners) or run them in a subagent. Massive log outputs get permanently added to your conversation history. 5. 🔍 Audit your /context: Run this in a fresh session to see exactly what's loaded from your CLAUDE.md or MCP tools, and trim the excess. 6. 📦 /compact before you step away: Prompt caches expire (after an hour on a subscription, or just 5 minutes on an API key). Summarizing your conversation is significantly cheaper while the cache is still warm. The main takeaway? Being efficient with tokens doesn't mean using fewer of them - it means ensuring every token goes toward the problem you're actually trying to solve #ai #programming #softwareengineering