Testing the Intelligence of your AI
1 like•202 views
Exactpro is a firm specializing in functional and non-functional testing for financial market infrastructures, having grown from 10 employees in 2009 to over 550 currently. They emphasize the increasing application of AI and machine learning in finance, potential biases, and testing intelligence in AI systems. The company completed a management buyout from the London Stock Exchange Group in January 2018 and is headquartered in the UK with operations in the US, Georgia, and Russia.
Testing the Intelligence of your AI
- 1. Testing the Intelligence of your AI Iosif Itkin, CEO and co-founder Elena Treshcheva, Researcher Iosif Itkin, CEO and co-founder Elena Treshcheva, Business Development Manager and Researcher
- 2. Exactpro Overview ● A specialist firm focused on functional and non-functional testing of exchanges, clearing houses, depositories, trade repositories and other financial market infrastructures. ● Incorporated in 2009 with 10 people, our company has experienced significant growth and is now employing over 550 specialists. ● We were part of the London Stock Exchange Group (LSEG) from May 2015 till January 2018. Exactpro management buyout from LSEG was successfully completed in January 2018. We are headquartered in the UK and have operations in the US, Georgia and Russia.
- 4. AI-based Systems in Finance Machine Learning in financial organizations: - already passed an initial development phase - the usage of live ML applications is about to dramatically increase over the next three years https://www.bankofengland.co.uk/- /media/boe/files/report/2019/machine-learning-in-uk- financial-services.pdf
- 5. AI-based Systems in Finance Machine Learning in financial organizations: - already passed an initial development phase - the usage of live ML applications is about to dramatically increase over the next three years ● Market Surveillance Systems ● Conversational Assistants ● Algo Trading Systems ● Pricing Calculators ● Machine Readable News ● Insurance Claims https://www.bankofengland.co.uk/- /media/boe/files/report/2019/machine-learning-in-uk- financial-services.pdf
- 6. AI-based Systems’ Quality Characteristics: - Ability to learn: The capacity of the system to learn from use for the system itself, or data and events it is exposed to. - Trustworthiness: The degree to which the system is trusted by stakeholders, for example a health diagnostic - Ability to generalize: The ability of the system to apply to different and previously unseen scenarios. A4Q AI and Software Testing Foundation Syllabus https://www.gasq.org/en/exam-modules/a4q-ai-and-software-testing.html Testing the Intelligence of your AI
- 7. Ability to Learn: https://www.deeplearning.ai/ • Training set — Which you run your learning algorithm on. • Development set — Which you use to tune parameters, select features, and make other decisions regarding the learning algorithm. Sometimes also called the hold-out cross validation set. • Test set — which you use to evaluate the performance of the algorithm, but not to make any decisions regarding what learning algorithm or parameters to use.
- 8. Trustworthiness: https://innovation.defense.gov/ai/ During the DIB’s quarterly public meeting on October 31, 2019, the DIB members voted to approve the proposed AI Principles.
- 10. Ability to Generalize: Scope of End-to-End and Negative Testing
- 11. Congruence bias Confirmation bias Law of triviality Zero-risk bias Anthropocentric thinking Illusion of control Cognitive Biases Affecting Software Testing of AI-based Systems Automation bias
- 12. AI-based Systems: Machine-Readable News
- 14. Salman, I. (2016). Cognitive biases in software quality and testing. Proceedings of the 38th International Conference on Software Engineering Companion - ICSE ’16. Pp. 823-826.
- 15. Mohanani, R., Salman, I., Turhan, B., Rodríguez, P., & Ralph, P. (2018). Cognitive Biases in Software Engineering: A Systematic Mapping Study. IEEE Transactions on Software Engineering
- 16. AI-based Systems: Conversational Assistants (Chatbots) Chatbot
- 17. Anthropocentric Bias We should not humanize computers.
- 18. Anthropocentric bias They dislike it a lot!
- 19. Anthropocentric Bias: Testing a Mine-Defusing Robot
- 20. Anthropocentric Bias: Why We Treat Robots Like Humans Darling, Kate and Nandy, Palash and Breazeal, Cynthia “Empathic Concern and the Effect of Stories in Human-Robot Interaction” (2015). Proceedings of the IEEE International Workshop on Robot and Human Communication (ROMAN), 2015. 6 p. https://www.ted.com/talks/kate_darling_why_we_ha ve_an_emotional_connection_to_robots
- 21. Anthropocentric Bias: Testing Chatbots Anaphora / Context Human: I bought 500 Company X shares two years ago. The stocks’ cost was 60,000 USD. What’s their today’s cost? Chatbot: What currency would you like to have for the rate? X Spelling / overall correctness Human: What is the setlement date of the tradeId XXX?? Chatbot: ???
- 22. AI-based Systems: Algo Trading
- 24. Indirect Testing Methods Information extraction and Machine learning End-to-End Automated Test Library Whatever it takes! Test execution data and log analysis Passive Testing Whatever it takes!
- 25. Applications of the Proposed Approach https://unsplash.com/search/photos/san-francisco The First IEEE International Conference on Artificial Intelligence Testing (IEEE AITest 2019), April 4-9 2019, San Francisco East Bay, CA, USA User-Assisted Log Analysis for Quality Control of Distributed Fintech Systems Iosif Itkin, Anna Gromova, Anton Sitnikov, Rostislav Yavorskiy, Evgenii Tsymbalov, Andrey Novikov and Kirill Rudakov.
- 26. AI-based Systems: Pricing Calculator
- 27. Law of Triviality (the Bike-Shed Effect)
- 28. Automation Bias
- 29. AI-based Systems: Fraud Detection and Market Surveillance
- 30. Build Software to Test Software Click to know more about Exactpro Test Tools
- 31. AI-based Systems: Insurance Claims
- 32. Zero-Risk Bias
- 33. Non-deterministic Systems: Financial Market Infrastructures
- 34. The Illusion of Control and Happiness Sherman, G. D., Lee, J. J., Cuddy, A. J. C., Renshon, J., Oveis, C., Gross, J. J., & Lerner, J. S. (2012). Leadership is associated with lower levels of stress. Proceedings of the National Academy of Sciences, 109(44), 17903–17907.
- 35. Fenton-O’Creevy, M., Nicholson, N., Soane, E., & Willman, P. (2003). “Trading on illusions: Unrealistic perceptions of control and trading performance”. Journal of Occupational and Organizational Psychology, 76(1), 53–68. The Illusion of Control and Performance
- 36. Thank you