Sylvia Ng

In my last article on how to set up analytics for your startup to scale, I left off on the topic of how important it is to measure your analytic team’s success. But measuring isn’t necessary an easy thing. Here’s a quick dive into some of the metrics that I considered using (based on my time at 500px when I wrote this article):

• # Data Warehouse Queries: Easy to define and quantify, but queries aren’t necessarily correlated with insights being generated — queries could increase for other reasons like low quality data being added to the warehouse, or poor warehouse design
• # Questions Answered: Directly related to what you want to be solving — answering people’s questions with data, but hard to implement and define
• Size of Warehouse: Easy to implement and track, but warehouse growth doesn’t necessarily mean data is being used or that insights are being generated
• # Dashboards Being Used: Dashboards indicate warehouse data is useful, but again, this metric is hard to track, and dashboard views may not be be directly correlated with data usage
• Analytics Satisfaction / NPS: Easy to implement through internal survey, metric can cover all facets of the analytics program, but subject to bias
• # Data Warehouse Users: Easy to implement and track, but doesn’t reflect how much each user may be utilizing the warehouse for insights, only that there are users
• # Data Warehouse Sections Under SLA: Great metric if you want to focus on data deliverability and reliability, but SLAs may be hard to put in place

From a technology perspective, the # of SLAs, up time of warehouse, #queries run, size of warehouse, etc were all good metrics but each had its own issues in terms of really representing analytics success at 500px. On the business side, we also considered # of analyses produced, and # of analysts hours logged toward certain initiatives, but these are extremely tricky to define and to measure properly we’d need to spend more time with logging then may be worthwhile. Beyond that, many of these measures can be easily gamed by the team.

But if the whole point is to make people empowered to do fact-based decision making, then an internal NPS score could make a decent measure of success. We implemented a first survey last quarter, where the analytics satisfaction score was measured on a scale of 5. Within that same survey we also asked for feedback on our analytics tools and what users need support on, thereby getting a sense of what we need to be working on keep increasing our analytics capabilities as a company. We’ll see soon enough whether our score improved for the quarter, with our latest survey.

Lastly, since analytics is embedded within Growth at 500px, a large part of the analytics team will be measured on our Growth KPIs… which is another topic for another day.