Responsible/Trustworthy AI in the Era of Foundation Models

Editor's Notes

• #3: Not AI algorithms and models Functional and non-functional requirements AI alignment + existential risks; AI safety; ethical/law risks
• #4: Entanglements, Cascades, Dependency, Unstable Data Dependencies, Hidden Feedback Loops Debt: Abstraction, Reproducibility ”Federated data collection, storage, model, and infrastructure” Interaction with other teams “co-design and co-versioning”…
• #8: Mechics/physics Bridges and buildings Fully understand the human brain to trust No Empirical software engineering and testing. Level of understanding ; I am not talking about you fully Why? My wife expecting, apology
• #11: Governance to connect with management Process to connect with other practices
• #14: The "science" suffix of computer science has sometimes been questioned and caricatured; perhaps not any longer, as AI becomes an ersatz natural science studying large learned artifacts. Likewise, LLMs are produced by a relatively simple training process (minimizing loss on next-token prediction, using a large training set from the internet, Github, Wikipedia etc.) but the resulting 175 billion parameter model is extremely inscrutable. This is the why the field of “AI interpretability” exists at all: to probe large models such as LLMs, and understand how they are producing the incredible results they are producing. Increasingly, the study of these large trained (but un-designed) systems seems destined to become a kind of natural science, even if an ersatz one: observing the capabilities they seem to have, doing a few ablation studies here and there, and trying to develop at least a qualitative understanding of the best practices for getting good performance out of them. Modulo the fact that these are going to be studies of in vitro rather than in vivo artifacts, they are similar to the grand goals of biology, which is  to "figure out" while being content to get by without proofs or guarantees. Indeed, machine learning is replete with research efforts focused more on why the system is doing what it is doing (sort of "FMRI studies" of large learned systems, if you will), instead of proving that we designed the system to do so. The knowledge we glean from such studies might allow us to intervene in modulating the system's behavior a little (as medicine does). The in vitro part does, of course, allow for far more targeted interventions than in vivo settings do. AI's  turn to natural science also has implications to computer science at large–given the outsized impact AI seems to be having on almost all areas of  computing.  The "science" suffix of computer science has sometimes been questioned and caricatured; perhaps not any longer, as AI becomes an ersatz natural science studying large learned artifacts. Of course, there might be significant methodological resistance and reservations to this shift. After all, CS has long been used to the "correct by construction" holy grail, and from there it is quite a  shift to getting used to living with systems that are at best incentivized ("dog trained")  to be sort of correct—sort of like us humans! Indeed, in a 2003 lecture, Turing laureate Leslie Lamport sounded alarms about the  very possibility of the future of computing belonging to biology rather than logic, saying it will lead us to living in a world of homeopathy and faith healing! To think that his angst was mostly at complex software systems that were still human-coded, rather than about these even more mysterious large learned models!
• #18: Everyone is a requirements engineering, architect and tester/verifier.