Systalyze's ML platform boosts Visa's AI performance

Transcript

The task that we have in front of us is to leverage our transaction data into a foundation sized and foundation capable model. This is a big task, so we brought in big help. The work that I'm about to describe is a collaboration between Visa and MIT founded startup Systemize. To understand what we did, start out by thinking about training a huge model in this manner. Which is pretty normal way to train a model. There's some research scientist who's trying to construct their model. The model architecture is one factor they must consider, but they must also consider resource availability. So like how many GPU's are there? What generation are they? How much money, how much memory do they have? How are they networked? They must also consider the software packages that they have running on there. I imagine many of you in this room have had moments where you've gone like this because of something that was happening with Torch. Vision. They have to consider whether the software packages are optimized for the hardware that they're running on, and they have to consider all of the other model hyperparameters on top of that. In most organizations, the tuning of this kind of system is essentially manual. The research scientist just kind of keeps all this in their head and tries to optimize it by hand. That process is wasteful. It wastes GPU cycles when the architecture and the software are not optimized for the available resources. It wastes engineering time exploring the hyperparameter space. And engineers and GPU cycles are both really expensive. So this waste, by our calculation, can come out to be in the 10s of millions of dollars. And envisage case, actually solving the cycle is extra hard because the hyperparameter space for transaction data is not well understood in the literature. So to say that another way, I, as a person who has worked on machine vision models a lot, have kind of a reasonably good sense in my head of the feel of the hyperparameter space for, say, building a unit. I kind of know like about how much data I need distributed, in what manner, what kind of learning rate, how long to train, how to perform the splits. All of that kind of stuff. But I know that because I and others have built so many units over the years, and I've done it a million times and I can read about all the other people that have done it a million times. I can read about, you know, if my loss isn't descending. I can read about someone who isn't exactly the same case as me and how they solved it. But no one has built a transaction foundation model because no one else has visas data set. So we're starting from scratch on understanding the feeling of this very complicated, multidimensional. Space. The intuition that could guide a lot of the manual hyperparameter tuning is absent. So we need help to be able to do this, and we're gonna use software to help us. If we imagine a theoretical software solution that could help us, let's call it a life cycle controller, it would need to have these features. It would need to help us manage the rapidly changing state-of-the-art software environment. It would need to help us keep the hardware optimized for best performance, for example by avoiding power overdraws. And it would need to provide a lighthouse for our engineers so they know where they are in the hyperparameter space. And how their model is doing versus how it could be doing theoretically. We've been very fortunate to be design partners with Crystallized to build this product out. And it worked really well. I'm showing you now data from a POC we did last summer. The measure here is training iteration time for an LM. We trained the model and based Kubernetes on Cubre and using the run dot AI environment and separate runs. The orange bar is our original performance. The blue bar is performance using the systemized performance controller. You can see that we were able to 8X our performance over the base Kubernetes and ray deployments and get almost that improvement even in the run dot AI environment as well. Amortized over an entire year, we estimate that this trading time benefit is equivalent to millions of dollars of sweet hours. Or if you don't like the sweet hour estimate, imagine at least that we can train 8 times as many models now in the same period of time on the same hardware as we could before we optimized the life cycle of this development.