Best practices for building machine learning platforms on the cloud
The evolution of enterprise platformsIncreasingly, organizations are becoming more adept at delivering top-notch customer experiences by leveraging cloud-native platforms like Kubernetes that can run large AI and ML workloads. Capital One’s move to become the first U.S. financial institution to go all in on the cloud and our ability to re-architect our data environment has been integral to expanding our cloud-based platform capabilities. With that strong foundation, we’re better able to leverage big data to build new ML capabilities on top of our enterprise platforms to accelerate, enhance and deliver on new, more meaningful customer experiences. Much of our work in this area is already showing impactful results for the business and for our customers. For example, our fraud decisioning platform was built from the ground up to make complex real-time decisions. By leveraging massive amounts of data and enabling model updates in days (as opposed to months), the platform helps protect millions of customers from card fraud and can be used by various stakeholders across the enterprise. Based on my experience leading teams to deliver enterprise technology platforms, there are important lessons and best practices I’ve learned along the way:
- It all starts with the team: Build a cross functional team of the best people, even if it slows you down at first. A bigger team is not always better! At minimum, the team should have product managers, engineers, and designers. Staff these functions with people who truly understand the users of the platform. For example, if you’re building a platform that will be used primarily by data scientists, hire a product manager who used to be a data scientist or put a data scientist on your leadership team. If the team is made up of people from several organizations, make sure you have shared goals.
- Work backwards from a well-defined end state: Before you start to build, take the time to align on the end state architecture and your plan to iterate your way to that destination. Make sure your architecture is designed for self-service and contribution from the start. Better yet, design the platform assuming that you will expand it to users outside of your immediate organization or line of business. Assume that over time you will want to swap out components as technology changes.
- Estimate how long you think it will take, then double it: It is important to take the time to brainstorm all the capabilities that you need to build at the outset and then create a t-shirt sized level of effort for each component. Once your tech teams marry this with velocity to estimate how long it will take to build each feature, add a 50% buffer. In my experience, this estimate ends up being surprisingly accurate.
- Focus on business outcomes: Building great platforms can take a long time. It is important to sequence the work so that business value can be achieved along the way. This motivates the team, builds credibility, and creates a virtuous cycle.
- Be radically transparent and over communicate: Share decisions, progress and roadmaps with stakeholders liberally. In addition to articulating what you are working on, also articulate what you are not currently prioritizing. Invest in documentation which enables contribution as well as easy onboarding to the platform.
- Start small: Even the best testing and QA environment can miss issues which are not found until something is put into production. For big changes that will have meaningful customer impact, always start with a tiny population, and then ramp up once you see things working in production at a small scale. When possible, use associates only for the initial population when a change impact external customer.
- Get serious about being well managed: Platform owners should obsess about platform performance. All issues should be self-identified through controls and automated alerts. Exceptions should be addressed quickly. Root cause analysis of issues as well as changes to prevent recurrence should be prioritized. A lack of issues should be properly celebrated so that teams know it is appreciated.
- If it seems too good to be true… Exception monitoring is a great way to ensure that your execution matches your intent. Often the goal is to have zero exceptions. For example, latency should never exceed 200 milliseconds. If your exception reporting NEVER shows any exceptions, it’s possible that the monitoring is broken. Always force an exception to make sure that it triggers properly. I’ve learned this one the hard way.
- A happy team is a productive team. Celebrate accomplishments, recognize team members when they go above and beyond and create a psychologically safe environment. Measure team happiness (with a quick 1-5 scale) regularly and give teams the space to discuss what would make them happier and the autonomy to try things out to squash dis-satisfiers.
10 Steps for Legacy IT Modernization Success
IT Modernization Success Although the thought of mainframe migration can be intimidating, it is a crucial step that businesses must
Accelerating AI for Growth – Infrastructure Plays a Key Role
Businesses all over the world have realised how important artificial intelligence (AI) is to driving change and company expansion. Many
Here’s Why Edge Computing Will Perfectly Size Its Role in 2023
With edge computing, it has always been possible to leverage “big data” (a term we now hardly ever hear) more
The 7 key trends for the security industry in 2023
In recent years, the physical security sector has seen significant change. In this constantly changing business, cutting-edge technology advancements and
See More Blogs
Machine Learning Helps Differentiate Compostable from Conventional Plastic Trash with “Very Good” Accuracy
Compostable plastics are becoming more popular, and while they have many advantages, some of these items, such as wrappers and
Big Data in B2B Business
Key takeaways Big data analytics gathers masses of data from numerous sources and uses it to enhance customer service and
Definition of programming A computer program is made up of code that is run by the computer to carry out