Best practices for building machine learning platforms on the cloud

The evolution of enterprise platforms

• It all starts with the team:  Build a cross functional team of the best people, even if it slows you down at first. A bigger team is not always better!  At minimum, the team should have product managers, engineers, and designers. Staff these functions with people who truly understand the users of the platform. For example, if you’re building a platform that will be used primarily by data scientists, hire a product manager who used to be a data scientist or put a data scientist on your leadership team. If the team is made up of people from several organizations, make sure you have shared goals.
• Work backwards from a well-defined end state: Before you start to build, take the time to align on the end state architecture and your plan to iterate your way to that destination. Make sure your architecture is designed for self-service and contribution from the start. Better yet, design the platform assuming that you will expand it to users outside of your immediate organization or line of business. Assume that over time you will want to swap out components as technology changes.
• Estimate how long you think it will take, then double it: It is important to take the time to brainstorm all the capabilities that you need to build at the outset and then create a t-shirt sized level of effort for each component. Once your tech teams marry this with velocity to estimate how long it will take to build each feature, add a 50% buffer.  In my experience, this estimate ends up being surprisingly accurate.
• Focus on business outcomes: Building great platforms can take a long time. It is important to sequence the work so that business value can be achieved along the way. This motivates the team, builds credibility, and creates a virtuous cycle.
• Be radically transparent and over communicate: Share decisions, progress and roadmaps with stakeholders liberally. In addition to articulating what you are working on, also articulate what you are not currently prioritizing. Invest in documentation which enables contribution as well as easy onboarding to the platform.
• Start small: Even the best testing and QA environment can miss issues which are not found until something is put into production. For big changes that will have meaningful customer impact, always start with a tiny population, and then ramp up once you see things working in production at a small scale. When possible, use associates only for the initial population when a change impact external customer.
• Get serious about being well managed: Platform owners should obsess about platform performance. All issues should be self-identified through controls and automated alerts. Exceptions should be addressed quickly. Root cause analysis of issues as well as changes to prevent recurrence should be prioritized. A lack of issues should be properly celebrated so that teams know it is appreciated.
• If it seems too good to be true… Exception monitoring is a great way to ensure that your execution matches your intent. Often the goal is to have zero exceptions. For example, latency should never exceed 200 milliseconds. If your exception reporting NEVER shows any exceptions, it’s possible that the monitoring is broken. Always force an exception to make sure that it triggers properly. I’ve learned this one the hard way.
• A happy team is a productive team. Celebrate accomplishments, recognize team members when they go above and beyond and create a psychologically safe environment. Measure team happiness (with a quick 1-5 scale) regularly and give teams the space to discuss what would make them happier and the autonomy to try things out to squash dis-satisfiers.