In this interview with Elijah Meeks, you can learn about how Netflix is analyzing, visualizing, and telling stories with massive datasets to drive intelligent decision-making within the organization.
- Thank you, Bill, thanks for inviting me. I'm happy to talk about data visualization anytime. - Excellent, so the focus, the theme for today's talk is really talking about big data, which, of course, is a big term And then there's this very practical and also social layer to it. So, the first one I'd like to focus in on is how, I build custom data visualizations. So, a lot of folks use tools to drive data visualization, and those tools can span from sort of GUI-driven, off-the-shelf tools, or a wide variety of libraries. And those are typically optimized for smaller datasets. And just like big data, small data has changed in its meaning over time. But tens of thousands of rows max, typically. So, oftentimes, I'm presented with a custom data visualization ask, that it isn't based at all on whether or not they need some kind of fancy, weird chart that you could only do in custom data viz, or some kind of very involved design process; but rather, they're coming to me and saying, "Hey, this data, we've got this dashboard in Tableau. "We've built it with x dataset, and it's great. "But now we need to drive this dashboard "off of a much larger dataset, "and we can't build a Tableau extract based off of that." And all they want from me is to build the same Tableau dashboard, but with custom data visualization, because then they can get access to some kind of these big data back ends, like Snowflake or Druid. We use Druid a lot for our big data. And so, it's very interesting to me that custom data visualization, especially when I started at Netflix, but even still, years later, is oftentimes framed as a data access solution, that if we could, we'd just, if there was a working Druid connector in Tableau, we would never have even hired you, right? - Right. - And so, I use that opportunity to build for them what they want, which is this very interesting situation because they have in their minds not just a view into the data, but a very specific visual representation of that view that they expect me to produce. So, if their Tableau dashboard had a pie chart and a bar chart and a line chart on it, and I came back to them with a cool Sankey diagram and a, I don't know, table full of sparklines, then typically they'd feel a certain sense of betrayal, right? Because implicit in their ask about visualizing this big data repository is that it would take the form of the visualization of whatever the sample or smaller subset of that data did. essentially going into the third wave, which is all about convergence. And you also, the second wave is also about developing tools. So, convergence is where we at, or where we're at now. Can you just describe what you mean by that, and what it means for what's coming next in data visualization? - So, when I was growing up, doing data visualization 10 years ago, the reason why one would choose to learn D3 or encode things was because you literally couldn't get certain data visualization forms, outside of specialized tools. And so, the only way you could get access to those was to learn these low-level geometric libraries that were all based off of the grammar of graphics or a similar systematic approach to encoding data channels with graphical, graphical attributes. That's no longer the case. You can do incredible things in Tableau. Just take a look at all the Tableau Public examples. There are amazing capabilities in a lot of different packages. Library-wise, you can do amazing, animated, graphically rich data visualization in R now in a way that you could only do in D3. Likewise, whether it's a BI tool or a custom application or in the notebook environment, all of the different modes that we're in are getting closer and closer to each other. So, there's no longer sort of this discreet thing called a dashboard, and it's very different from something called a report, and it's very different from something called a notebook. Instead, they all share all the same capabilities and the same expectations among their audiences. And so, I think right now we're all dealing with this in a very reactive kind of way. We're responding to these changes intuitively. But we haven't sort of actively and explicitly called it out and theorized about what it means for us that all of these things, all of these tools now have very similar capabilities. All of these modes have very similar users and audience expectations. And all of the ways that we're presenting things share in common a lot of the same forms, so that you have interactive elements in data-driven storytelling in journalism, and you have a lot of journalistic elements and aesthetic pop in internal business development applications. And so, I don't have necessarily the most concrete answers for what to do now, other than a few things that I touch on in a couple of my talks; for instance, saying that even if you're developing business applications in industry, you should acknowledge that it's an attention economy and you still have to draw your users in, you still have to use techniques to engage your users and draw them into your data visualization because the dashboard or report you make is one of many that they're going to have in front of them. And so, you have to make yours stand out in the same way that the New York Times has to make their stories stand out. But otherwise, I think it's more an acknowledgement of this shift and really struggling with that and ideating around that to try to figure out what that means for the future of data visualization when it's no longer technically challenging to produce things, and instead, now, it's the question of what tool you make or what profession you're in, or what audience you're building for no longer constrains your choices. - Yeah, yeah, that's great. I mean, what I always say is that as the things get more and more commoditized and simplified and the technical solutions go away, it all comes back to the idea. It all comes back to that - That's right. - communication strategy. What am I tryin' to say? Who am I talkin' to? What do they need to hear? And so, it's always there first. And if you come up with those ideas first, then the technical solutions melt away, and more so now, as you said, than ever before. - Absolutely, and I think that's why we need this emphasis on design and an emphasis on storytelling, and really, especially for the sort of C-suite audience for things like this, an emphasis on why data visualization and what data visualization is most impactful, so that they can know how to invest and how to evaluate it to know whether or not their organization is well equipped to enter a world where data visualization is everywhere and key to the success of an organization. - Absolutely. Well, that's unfortunately all the time that we have. Elijah, I want to thank you very much for joining me today. We dipped our toes in the shallow end of the very deep pool of big data. There's so much more to talk about, but we'll have to tackle that another day. But thank you very much for joining me and giving us your time. - Bill, can I make one pitch for the Data Visualization Society? - Sure, go for it. - Yeah, I'd love for the listeners, if you're a data visualization practitioner, and that is broadly construed, and then I would like them to take a look at DataVisualizationSociety.com, which is a new and extremely active professional organization focused on holistic data visualization, so not focused on a particular tool or a particular mode of data visualization. And we have an excellent publication and a growing member list and an extremely active Slack. If you would like to join, it's free. And we've got a lot of opportunities for uplifting voices and improving your profile in the field. - Yeah, and I have to say, I'm a member of it, and I am on the Slack community, and it is a very vibrant community. So, absolutely, great job. - Thank you, Bill.