The Tools Early-Stage Startups Actually Need to Understand Their Customers
Management

The Tools Early-Stage Startups Actually Need to Understand Their Customers

Segment Co-founder and CEO Peter Reinhardt has personally test-driven a tool a week for the last three years. Here he shares the toolkit you need to get the most out of your customer data.

After 18 months of struggling to get traction with two product ideas, Segment started its turnaround. What followed was a two-year stretch of growth from four to 60 people, thousands of new customers and $44 million over several rounds of financing. What was the trigger for such a significant inflection point? In the lead up to the sea change, Co-founder and CEO Peter Reinhardt and the team at Segment started to heavily lean into qualitative feedback tools like live chat widget Olark. It was embedded on each page and everyone from the newest hire to the co-founders were constantly learning from everyone using their product.

It’s hard to pin all this momentum on tools, but knowing how to leverage them is at the core of Segment’s business. The startup makes it easy to collect customer data and send it to any tool for any purpose, from analytics to marketing automation. That’s why Reinhardt has become intimately familiar with every tool that he can find that enhances customer data, personally test-driving about a tool per week for the last three years. Segment has integrated hundreds of tools to help over 6,000 customers, including Product Hunt, Atlassian and Instacart, build and refine their toolkit.

In this interview, Reinhardt draws from his extensive evaluation of tools to share how and when resource-constrained startups should select them. Here, he weighs in on the build-versus-buy debate, provides recommendations on which tools to consider across eight distinct categories, outlines the pitfalls to avoid when choosing tools and when to declare tools successful.

Where to Start When Selecting Tools

When most people want to make use of customer data, they think of analytics, but advertising tools, help desks, CRMs, attribution and a dozen other categories of tools are also powered by customer data. “What all these tools have in common is that they use customer information and behavioral data to help business teams improve the customer’s journey,” says Reinhardt. “When we think about customer data, that doesn’t include the information that funnels into HR or accounting tools. That’s because customer data is ultimately behavioral data: knowing when a user has viewed a page, purchased a product or invited a friend. It’s defining and refining who a user — or customer — is by the many actions they take.”

First decide to build or to buy.

It’s impossible to operate a business without interacting with customers and capturing their profiles, especially in the early, formative years of a company. “If you're building a web or mobile business, the way in which you interact with your customers is often via e-mail, push notifications, SMS or surveys inside or outside the product,” says Reinhardt. “The way you discover how easily they're using your product is with an analytics tool that shows you a funnel. The brick and mortar analog is a store owner standing in her store observing what's happening. It’s that fundamental.”

So, one of the first questions that arises is whether to purchase off-the-shelf or in-house tools. The former is ready to use immediately, but likely costly and not precisely calibrated to the business. The latter can tie up precious engineering resources and risk reinventing the wheel. “This decision actually has less to do with growth stage. Many companies mistakenly build tools for themselves — instead of focusing entirely on their customers — because they understand themselves better than their customers. It's an easy trap,” says Reinhardt. “You should only be building your own tool if you've tried lots of others on the market and find that none of them will solve your problem. Only build if you’re left without a choice. And still take care: I’d estimate that 50% of the startups that I see build tools can’t maintain them.”

If you’ve done your research and decide to build, go in with eyes wide open. “Building your own analytics tools in lieu of unleashing all your resources to build your product can be dangerous. It can easily kill a startup,” says Reinhardt. “Say you have three engineers and one of them is working on building your analytics tool. Your ability to develop your product is now cut by 30% so you need 30% more runway. That analytics tool is lot more expensive to build than it seems. Being capable of building something doesn't mean that you should.”

Compile a comprehensive checklist of tool categories.

If you decide to test and buy tools, it can be challenging to untangle what to select, let alone the universe of tool types out there. Reinhardt has identified eight categories of tools, which should serve as a sufficient starting checklist for most startups. Here are his categories and top recommendations:

Timing makes good tools great.

A big task in selecting the right tools is integrating them at the right moment. This may help keep you from buying expensive tools too early or continuing to use basic tools if you’ve outgrown them. Here’s a selection of Reinhardt’s recommendations as you act upon his tool checklist:

  • On deciding where to start. “As far as customer data analytics tools are concerned, by and large startups begin with Google Analytics. It may seem obvious, but don’t overlook it. It’ll give your startup a basic idea of your visitors, what content they’re viewing and where they are coming from. It’s free, a pretty good tool and will get you through the first year in terms of analytics for the most part. You’d have to be crazy not to turn that on.”
  • On upleveling analytics tools. “Start integrating tools for more complex analytics after you get to product-market fit. When you’re getting more granular and want a deeper understanding of, say, the conversion funnel from the homepage to the innards of your product, expand to a full funnel analysis tool like Amplitude, Mixpanel or Kissmetrics. You’ll then reach a point — likely around 30 people —- when the questions that you’re asking about your customer data become too complicated for an out-of-the-box tool. If you’re looking beyond the funnel to a more holistic analysis of your business, start considering Looker, Mode or Periscope to give you SQL Business Intelligence systems to get an operation-wide view. These tools connect to data warehouses like Redshift.”
  • On diving into diagnosis tools. “There comes a point when you’ve got a sense for where there’s a leak in the funnel, such as when people are signing up, but aren’t activating. At this point, we’ve found that qualitative feedback is much more interesting and impactful than quantitative analysis, especially when you have a statistically insignificant number of customers. Consider live chat tools like Olark, Livechat and Intercom. I'm not sure how you would even get started building a web or mobile product without some sort of really high bandwidth communication mechanism to your customers.”
  • On switching on visitor recording. “Visitor recording can be very helpful with product marketing and life cycle. It’s a valuable tool for product teams to use to understand what’s happening at the individual user level. So if there’s only 10% conversion at a specific step in the product activation flow, you can watch recordings of users’ behavior and see that the button is invisible, for example.”
  • On pursuing market attribution tools: “When a consumer startup starts investing in paid acquisition as part of its marketing efforts, the top of the funnel analysis from Google Analytics won’t cut it. Shop for a marketing attribution tool such as Convertro for web and Kochava or Hasoffers for mobile.
  • On introducing drip emails. “Introduce drip emails at the point when your analytics and diagnosis tools are revealing where users are dropping off, but you can’t yet get to it in the queue of product fixes. Send them messages to help them plug that hole or get them over a barrier that they might have encountered.”
  • On activating sales and support tools. “These products are truly beneficial after your analytics, diagnosis and drip-email tools are working together and there’s the beginnings of a sales and support team. So that they can do their job, they need more context about what the customer experience has been so far. That requires extracting data about funnels and actions customers have taken in the app so that sales and support people can help customers without constantly needing to ask what they’ve been up to.”

Switch between qualitative and quantitative tools with logic.

Reinhardt sees a common pattern among startups as they expand tools to understand the far corners of their business. “They tend to oscillate between quantitative and qualitative tools. Examples of the former are analytics, attribution or SQL BI tools, while the latter include live chat, visitor recording or survey tools,” says Reinhardt. “The quantitative tools tend to help funnel failures or successes, while the qualitative is diagnostic in nature and is valuable in revealing why it’s happening.”

Use quantitative tools to discover symptoms or opportunities and qualitative tools to diagnose them.

Startups should feel comfortable in a rhythm as they move between qualitative and quantitative tools. “That’s part of spiraling up. People will often switch back and forth between the two types of tools to improve,” says Reinhardt. “Don’t fight it. Make the switch as your focus shifts. If you look at the funnel and see a bigger drop than what you were expecting, step out of the analytics tool. Turn to your suite of qualitative tools — perhaps a live chat or visitor recording — to suss out why that may be. Test your hypothesis with a tweak to your process and check your analytics. Rinse, repeat.”

Pitfalls to Avoid When Choosing Tools

Even with a clear plan, Reinhardt continues to see companies make a few basic mistakes when selecting tools. Here are his top three lessons of caution:

Test-drive the off-the-shelf tool — don’t drown in research.

You’re seeking new tools to better understand or equip your company, but there’s no replacement for just trying it out. “I think it's really difficult to assess whether tool is a good fit before trying to use it and before playing around in the UI. Yet most people research because it’s onerous to fully integrate and run enough of your data through a system to measure its value,” says Reinhardt. “Well, not to be an ad, but that is where a tool like Segment is helpful. After you set up your tracking once, you can just flip switches to have it flowing through several tools. This lets you easily juxtapose them. Whether you use Segment or not, find a way to lower the bar for seeing your data in motion through a tool.”

People who are responsible for vendor selection want to show that they are running a thorough process internally. “The irony is that I've found that the companies that are trying to do a really good job in the vendor selection process shoot themselves in the foot. That’s because the ‘better' you do in the vendor selection process the longer it takes for you to try the tool,” says Reinhardt. “The success rate that I’ve seen with in-depth interviews with vendors is really low, which means the iteration time to try new tools is really long. Figure out a way to test them quickly.”

If it’s not vendor research, the most simple and impactful preparation is simply to write down the objective — even if it is in hypothesis form — before testing the tool. “Enter the tool selection process with an attitude of confidence that comes from experimentation. It may seem apparent, but I’m shocked by how many companies forget this step because the are rushing to make a decision,” says Reinhardt. “Here’s an example: ‘Our goal in using the analytics tool is for our product team to have a complete picture of the activation funnel.’ In this case, define what 60%, 80% or 100% of a complete picture of the customer might look like at various stages of success with the tool. Test it with your data, and, at the end of two weeks, declare whether it hit mark. This setup will help you find the winner or keep you moving.”

The other advantage to testing a tool is to gauge the level of support for the tool. “Interviewing vendors is obtaining a vision of what the product can and might do. Getting support during a trial is getting a sense of what the product does,” says Reinhardt. “Don’t forget the support experience. The tool’s support team should be there early to ensure your trial is successful. With tools there can be big barriers to switching platforms, so if they can’t serve you well during a short trial, that’s not a good sign for long-term support.”

Successful companies constantly experiment with new tools and cut bait if they don't work. They verify through short trials not a long research process.

Calculate the full cost of the tool.

Sticker price is only the beginning. Quantifying time saved and opportunity cost gives a more realistic approximation of the true cost of a tool. “There was a point in Segment’s history where we we had only three months of runway left. It was a very scary time and we ran through all of the scenarios to survive if we weren’t able to fundraise,” says Reinhardt. “Compared to people, tools were a small fraction, but we couldn’t part with either, even in our desperate situation. It was because tools allowed us to save time, and that was hugely valuable.”

Reinhardt and Segment compare the cost of a tool to the dollar value of their time. “Tools not only save us money on hourly cost, but also accelerate our work,” says Reinhardt. “Say you are assessing your time at $50 per hour and buying MailChimp at $100 per month will save you building an e-mail distribution tool. That’s clearly worth it. So for us, it was getting disciplined about measuring the tradeoff between our time and budget.”

That’s even before calculating opportunity cost. “In retrospect, it can be enormous,” says Reinhardt. “If the company succeeds then you're going to end up valuing your time a lot more than the $30,000 dollars a year you might be paying yourself. It’s harder to quantify given how highly variable it is, but it can tip the scale if you are making a 50/50 decision.”

If you are truly paralyzed in purchasing a tool or service, take solace in the fact that most contracts for these tools are far less aggressive than, say, your mobile plan. “I structure the deal with vendors to pay monthly, but pretty much all these vendors will do that by default, especially for a startup that’s pre-product-market fit,” As companies get bigger they’ll ask for annual contracts, but at that point, it’s not a big deal,” says Reinhardt. “The bigger deal is getting locked in; that is real. Say you start sending your data to a tool and six months later you discover you were using the wrong tool. What you actually need is that data in another tool. It's getting very difficult to untangle yourself, because you've installed tracking code all over the place.”

Switching cost is a variable to factor into the total expense of a tool. “Again, through the course of an early-stage startup, a company will use several tools in each of eight or so different categories. For example, your data must graduate from different analytics tools as your startup grows,” says Reinhardt. “Don’t get entangled in a tool that you don't want to use anymore or that your company has outgrown. One way to do that is to make sure that you are separating data collection from the tools that receive the data.”

Don’t overdose on quantitative data.

For early stage companies, capturing continuous customer feedback and perspectives is more important than detailed analytics. “Your near-singular goal at that stage is to iterate on the product itself or product positioning. Until you have proven product-market fit, the most important thing you can be doing is customer development,” says Reinhardt. “You probably don't have the volume for statistical significance at this point, so focus more on that qualitative component, essentially what customers want and or need.”

This lesson especially hit home to Reinhardt while he and his co-founders were building an earlier product at Y Combinator. “Our goal was to build a classroom lecture tool. On a basic level, students could push a button to say ‘It’s confusing’ to the teacher anonymously, and the professor could see a graph of how puzzled his students were,” says Reinhardt. “We spent 3 months at YC working on it. That meant hundreds of thousands of lines of code. The massive spread of product features produced a wide variety of data to analyze, from presentation slides, student note-taking and real time Q&A. We were obsessed with testing and tracking each classroom setting. But the critical insight came when we physically sat in the back of a classroom to observe what was happening firsthand to see what students were actually doing. Turns out, they were opening their laptops and going straight to Facebook.”

The moral of the story is to avoid seeing your product as a composition of metrics to track before you treat it as an opportunity to assess customer behavior. “The failure mode of the product or the business was really obvious. If we had just observed the students, we would have known instantly, but it was totally not obvious if only done in a quantitative way,” says Reinhardt. “The qualitative aspect was critical at that stage, when we had a statistically insignificant number of people using it. So, under no scenario should you get distracted by quantitative metrics that don't really matter — there’s a cost, and it’s usually at the expense of the qualitative data that’s most essential early on.”

How to Gauge the Success of Your Tools

Each company and category has its own needs for its tools, but Reinhardt believes that there is a general benchmark of success for each type of tool. “It can be tough to say generically across categories, so it’s helpful to have a general expectation,” he says. Here is his baseline for success for each tool:

  • Basic top of the funnel analytics: “For Google Analytics, it is knowing how much traffic you’re getting. You know which parts of your product landing pages are resonating, who in the ecosystem is talking about you and sending you traffic.”
  • Mid- to deep-funnel analytics: “For the event tracking tools, such as Mixpanel, Amplitude and Kissmetrics, you’re successful when you have a full funnel analysis for each major product work flow, so you know how likely a user will be to complete a major set of actions inside your product. You should be able to extract next steps on how to improve that process.”
  • Business-wide analytics: “For SQL BI analytics tools, such as Looker, Mode and Periscope, one should be able to build high-quality dashboards for every team, including the executive team and the board of directors.”
  • Diagnosis tools: “For chat tools, your product manager should feels as they're actually learning a lot. It’s a more qualitative assessment, so we institute a scale (1-10) to assess whether PMs find the tool helpful for building context about what customers need.”
  • Drip email: “For email tools, success is being able to capture a list in a funnel. You want to be able to bounce back and forth between Customer IO or Autopilot and a campaign, and see the conversion rate fluctuate in different parts of the funnel.”
  • Sales and support tools: “For tools such as Zendesk and Salesforce, success is your sales and support folks having complete context about a customer and giving them an exceptional level of service.”
  • Paid acquisition and attribution tools: “For paid acquisition or attribution tools, look to see if you’re actually acquiring customers through it and if you're able to measure how much it cost you to acquire each customer. Integrate this data with your analytics tool to see how much revenue that customer generated.”

Your expectation for each of these tool categories is that they are instrumented with your data and deliver insights that change how you make decisions. “That could be making choices faster, in a more informed way, or more collaboratively with your team or customers. If you have a tool running the background, but you’re never looking at it, it’s not valuable.”

Success comes from implementing a tool, using it and making decisions with the data it’s serving up to you.

It’s enough of a challenge to structure and disentangle data; there’s no reason to have the same complications with the tools you use to manage it. Use Reinhardt’s eight categories of customer data tools not only as a taxonomy, but also as a timeline to follow as your startup scales its operation and its data set. In most cases, default to experimenting with and buying your tools, which is a more holistic, cost-effective approach when startup resources and runways vacillate. Seek an equilibrium between tools that measure quantitative and qualitative outputs, and ensure they meet a predetermined, baseline mark of success for each type of tool.

“There’s a reason why customer data becomes a mess. It’s because customer data is valuable to so many parties. Each team — sales, marketing, product, design, the executive team, you name it — needs access to more and more data to do their job well,” says Reinhardt. “The truth is, that unlike many other contested resources, data has the potential to be used by all its suitors completely and simultaneously, if organized correctly. The key is to create a data repository that allows each tool to share one complete version of the data, which, when routed to each team’s tools, seems to have been collected especially for their purpose. That’s the magic.”