Michael Graham recalls the side project that launched his career in security. An analyst at a major transportation company, he installed a series of open source network intrusion detection devices — a brand new technology set at the time — and started to rebuild the underlying OS to get them functional. His goal was to capture enough traffic to more clearly diagnose recent network issues. Two weeks later, the infamous computer worm Code Red struck. While the technology that tracked packages and planes was intact, almost the entire corporate network went down. While the company began working on isolating and containing the problem, he started digging into his project’s nascent early logging system. He found the contractor laptop — sitting two floors beneath him — from which the initial infection originated.
What if that side project had been his full-time job? He could have saved the company a lot of headache and money. That’s when Graham knew that security isn’t peripheral; it’s a business critical function. To recognize and act quickly on threats is to determine whether a company can operate efficiently. To identify and minimize those dangers dictates whether a company can plan correctly. Over the last 15 years, Graham has held senior posts in security architecture at Harrah's Entertainment, Zynga, Evernote and Box, where he’s currently a Senior Manager of Security Architecture and Engineering. In his career, he’s led security efforts across stage and sector: from startups to Fortune 500 companies and from public education to manufacturing.
In this exclusive interview, Graham outlines the distinct ways that startups should approach security at each growth stage, including when they should make their first security hire. He also shares how to best position security in the broader organization and how he sees the function evolving over the coming decade.
How To Think About Security Spend as a Startup
Many resource-strapped startups gauge their commitment level to security by assessing the financial expense to the company. Instead, Graham recommends defining security spend by a company’s possible exposure risk. “For all companies, there’s a limit to how much money can be lost. So if you’re spending more than that amount, you’re absolutely screwing up,” says Graham. “There’s also a limit to how much money you’re likely to lose based on what it is you do with customer data and what you do to monetize it. You’re also messing up if you spend more than that amount.”
Graham admits that these assertions are counter to many marketing messages. Most startups are exposing customers to more risk than they’re selling. “There’s a lot of social capital used in marketing these days. Statements such as: ‘You can absolutely trust us to take care of your data.’ I’m sorry, but when you’re a six-person company, the reality is that’s just not the case,” says Graham. “You’re not checking everything. You’re not doing extremely deep code reviews. You’re not managing every possible, realistic threat. As a startup, you’re doing a whole lot of hard prioritization, but it’s unfortunately likely that security is not always ranked near the top.”
This reality is as much due to the challenging path of early-stage startups as it is to broader industry forces. “Company-stage circumstances aside, this reality is also a side effect of doing business in a venture capital-driven environment. You're trading your time and other people's money to acquire customers as fast as you can,” says Graham. “In this mode, customer acquisition can quickly slide into representing trust when trust is relevant. It's not about spending every possible dime you can to make that trust rational. Whether or not you agree with that stance, acknowledge that mentality as you build your threat models to protect vulnerabilities. Once you pinpoint those susceptible points, it’s easier to build controls to better guard them. Then, like everything else in the startup world, you must rationalize your spend for the impact.”
Security isn’t about reducing risk to zero. It’s about continuously rebalancing risk as context shifts.
To assess risk, begin by seeking an outcome that is a balance of tradeoffs that will be reevaluated as fast and frequently as the business evolves. “Every early threat is existential, so the only way a startup can operate without the paralysis of the unknown is to identify its point of tolerance on a sliding scale of threats,” says Graham. “You can be religious with a line-by-line code review on everything or spend hours reviewing every single package that goes into your OS or application containers. You can take security to the nth degree and burn any risk out. But you’ll also likely remove all the speed and profitability out the product as well.”
Given the sensitivity around the details of the specific cases from which he draws, Graham has abstracted how he’s seen the evolution of security for startups, not only from his point of view, but also from his peers’ perspectives in the field. From his vantage point, here’s how early-stage companies balance the risk and reward of security. Given the variety of growth trajectories for startups, he segments loosely by company headcount:
- 1-9 employees. “At this stage, avoid extra security measures that will bog you down. Use two-factor authentication. Don’t put your servers on the internet with all your services exposed. Put them instead on somebody else’s cloud, and make sure you’ve got your assets locked down to the point where they’re not interesting to opportunists. The odds are that no one is targeting you — or your potential customers. You're only going to have a real security problem if someone randomly happens across your infrastructure. Make that unlikely. Setting up shop on AWS is about a day's worth of work for your initial needs. Just do it and get back to growing your product.”
- 10-19 employees. “With a headcount in the double digits, you’re starting to divvy up responsibilities. You're past the point where everyone works on everything, but still at the stage where deciding to do something only requires a five-minute sync, not a team meeting. This is the stage when you want to have someone who’s responsible for security. It need not — and likely won’t — be her full job, but it must be in her purview. It’s likely your CTO or whomever might grow into that title. This person monitors infrastructure as part of her role, determines how code is pushed and knows where the secrets live. She’s not yet making the call on hard decisions about risk, because others are as familiar with the business, code base and infrastructure. But she’s able to take a defensive posture, ensuring that no one is doing anything that’ll attract unwanted attention. At this stage, the company is in a vulnerable, asynchronous relationship with anyone who threatens it. Attackers have infinitely more resources and time than the startup does to defend against breaches.”
- 20-49 employees. “If you’ve survived, you’ve got a product and are building relationships with customers. That means you’re their trust partner, whether you realize it or not. They’re giving you some level of information, maybe even about their customers, and you’re doing things with it. Now you not only have your possible threats, but also those of anybody who might target your customers. This is where it becomes dangerous, because if you’re working with interesting data, you’ve got the potential for real threat actors. Add in that you’re likely hiring engineers who can program, but may not know how to review that code or design a secure system. Lastly, now you’re starting to scale your infrastructure and its corresponding security controls. With this many moving parts, cracks start to appear. I honestly think a lot of companies get through this point only through luck. I'm not sure there's a better answer than that.”
- 50 employees. “You’ve made it to 50. There’s likely an advisor, co-founder or investor that’s thinking you might get to 100 employees in a year, if everything keeps going well. If you haven’t already, this is the time when you should bring on your first, full-time dedicated security hire. You’ll hear things like: ‘We only have room for 10 non-eng hires… and we need someone in legal, contracts, HR, etcetera.’ What usually happens is that a full-time hire isn’t made and the next inflection point is around 200 to 300 people. That’s the point when it feels like a ‘real’ company that’s going to work out. Then, people are hired for the big picture problems in ops, security, talent and more. But when that person comes in, they don’t just start from scratch — they acquire all the technical debt related to their discipline because no one was fully owning it before.”
Onboard your first, full-time security hire between 30-100 employees.
Graham provides a wide range not only to account for the different growth paths of startups, but also to counter his bias as a security professional. “Of course I encourage people to bring in the first security hire closer to 30 people than 100 employees. It’s because it’s at the point when the technology and business sides of a startup concurrently and exponentially grow faster and more complex,” says Graham. “The danger is that business leaders of an early-stage startup often see security as a technical problem and engineering leaders see it as an issue that can be resolved by applying their existing technical know-how and rigor. That’s a naive misstep.”
The reason to hire a security leader early stems from the same rationale as that of hiring any professional at a critical business juncture: as much for her ability as her judgment. “You’re hiring someone who’s gut you trust, not because she’ll join and start blindly slathering everything with security process and rigor,” says Graham. “You want her to talk to teams, look at code and tell you where you’re going to get burned. She should be able to demonstrate that she can protect the organization as it moves, not just batten down the hatches to secure the startup in a foxhole. Lastly, she should start fixing old security technical debt and intercepting and reducing new debt.”
Admittedly, the difference between a 30 and 100-person company is significant, and choosing the right time to make the first security hire is always clearer in retrospect. “If you’re doing something ‘high trust,’ such as storing someone’s tax documents or shipping a messaging client, hire closer to a 30-person headcount. If there are compliance requirements before releasing an MVP, that clearly puts you in this category,” says Graham. “Then you have companies for which security isn’t a part of the product, excluding how anything on the internet should have measures in place not to get hacked. These are companies that aren’t building a high-trust product. I’d still recommend that companies bring on the first security hire earlier than later, but these types of startups can get away with onboarding a security leader when headcount hits 100 employees.”
The reason Graham has honed this security hire timeline is because there’s danger when a startup becomes a “real company” and the foundation for security has not been set. “If you wait to build out security at that point, both the company and the first security hire are going to regret it. At one organization, Graham joined to help form a security team after a breach. During the interview cycle, Graham and a few other candidates recommended that they hire a team, not a single security hire,” says Graham. “The organization was past the point where one person could handle the challenges. It was generally a security conscious company and most of the right security measures and responses to the breach were done correctly. Fortunately, they got by with their engineering prowess, but their methods already revealed long-term security cracks.”
Security’s bias is avoidance of negative consequences. Engineers seek to optimize positive outcomes. Both may be technically capable of handling security instances, but eventually, their motivations will build starkly different secure systems. “In the case of the breach, a security person would’ve put in a system that would’ve decreased further risk. For instance, passwords could’ve been compromised after the breach. A security architect would’ve used a hashing mechanism to make a ‘second strike’ harder,” says Graham. “It would have saved both the time and resources of the engineering and customer success teams. Those teams could’ve more assuredly said: ‘We're reasonably certain it will take them months to break into these passwords. If you're using that password anywhere else go change it, but don't worry.’”
Wait six months too long to make your first security hire and the person will start with 1-2 years of security debt.
How to Smartly Position Security at Your Startup
It’s not uncommon for startups to associate security with a ‘big brother’ and those in security to feel as if they are babysitting employees. For many generations, security meant managing firewalls and any interaction with the function was largely concerning policy enforcement. Security became the de facto police for behavior at work, monitoring how you browsed the internet and what you installed, down to the letter of the company’s internal societal norm.
“Security as a concept in business is still recovering from that time period. Employees go in thinking of security as a blocker, a business complication and an expensive cost center. That is all somewhat true, but not the whole story,” says Graham. “Employees assume that security is in a room playing ‘big brother.’ I don’t know anyone who works on a modern security team who views themselves that way. That presumption is a holdover from older times.”
Given that bias, the responsibility of today’s security function — and a startup’s founders — is to evangelize its real role as a cooperative protector and a source of advice. “This is an uphill battle as security — and the startup at large — is strapped for resources and often playing catch-up. This often prevents them from proactively giving advice internally. They become more of a babysitter just catching vulnerabilities before they’re compromised,” says Graham.
Here are five ways that can help security move from big brother or babysitter to advisor:
Train colleagues to double-take on trust decisions.
Startups go to great measures to prompt specific behaviors in their users. The same approach can be applied internally when it comes to security measures. “The goal is not to get employees to do their own threat modeling. When it comes to risks, most people think about the worst incident that could happen, wish away the likelihood and proceed. A slight shift in perspective can make a big difference,” says Graham. “Instead, introduce one-more checkpoint to give an employee pause. One way to do that is to show employees their trust decisions — from a security standpoint — in their work. For an engineer, that could be validating that everything’s normal at a specific point of an authentication system. If all’s well, security should be OK.”
If employees can do even a quick double-take consistently, then the security team will have significantly fewer fires to put out. “If you can teach people to think twice at security crossroads in their own work, you can take two thirds of security problems and wipe them off the table,” says Graham. “People who normally trigger issues will implement a fix. Even if that fix is just spending 20 seconds flagging an email attachment that looks unusual. If an employee pauses because he knows that viewing a file is making a trust decision, security wins. Then it can focus on fixing highly technical and domain-specific security issues instead of firefighting.”
Here are a half dozen questions that employees can ask themselves in order to develop a habit or instinct of thinking twice:
- Was I expecting this email or phone call?
- How have I confirmed that this person asking me to do or share something is who he says he is?
- If I’m telling someone something sensitive, am I sure she’s supposed to know it?
- What assumptions have I made about what someone will do with what I’m building?
- If I’m touching how we manage secrets, authentication or sessions, have I asked someone for advice?
- If I’m wrong about this decision, how would we find out?
This line of self-inquiry won’t replace a security executive or team helping with threat modeling, implementing controls or even just offering quick advice, but it’s a broadscale way to help prevent major missteps. The thrust and theme behind the questions are always “Is this unexpected?” and “Am I making a big assumption?”
If you see something, say something. The PSA rings true for security at startups. It’s the best way to keep an employee from becoming patient zero.
Kill the shame game.
The primary way to encourage employees to reach out proactively to security is to immediately get out of the scolding cycle. “People won’t interact with you because they're afraid they're going to get told something they don't want to hear. They might be fearful that they’ll be admonished. This is not unique to security. It’s true of any consulting function inside a company, such as HR or legal,” says Graham. “When every interaction is high-stress, that builds a bad relationship — and one that will eventually become transactionally expensive. What it comes down to is: don’t scold or embarrass people. Shame won’t work as a behavior changer. It will make everything you’re trying to avoid more likely to happen.”
Reward detectives.
Many companies have programs that recognize employees who go above and beyond in their jobs — security should do the same. “Whether it’s as easy as an email shout-out or recognition at an All Hands, find a low-lift way to acknowledge employees that flag security issues. The hat tip — whether done publicly or privately — should mirror your company’s culture,” says Graham.
“At Box, we have what we call the security star program. We dole out physical stickers and are now working on a new trophy as the prize, because it’s a big win when people take a moment to give us a heads up,” says Graham. “If it’s a phishing campaign that’s gone out to a group of people at the company, we’ll credit the employee who caught it. We highlight the person and thank them for notifying us so fast. Employees are our number one source of detection."
Security-wise, I don't need to stop every bad thing that happens. I need to have enough detection capability and colleagues who’ll notify me so we can react in time.
Forget education, emphasize email.
Other avenues to train employees include educational methods, such as modules, handbooks or recurring security digest meetings. Graham is less bullish on those tactics, and while compliance often pushes these projects forward, he doesn’t suggest making them a key pillar of your defenses. “I've never seen a security education program that actually works — at least not one that improved independent decision making in its participants,” he says. “In the end, you want an open, comfortable channel. The education that works is getting people to email security. You need everyone in the company to know that — for a fact — emailing security about something is never the wrong thing to do. Your ‘thank you’ is having a process that responds quickly.”
All Hands and email blasts aside, the method that has proven most successful for Graham is his sign-off and quick response to an incoming note. “Keep messages short and simple; it’s most important how you sign-off. ‘If you're not sure just email security’ should go at the tail end of nearly everything you ever say to anyone. I don’t make it an email signature, but type it out differently each time I think about it, so it’s authentic and not automatic,” says Graham. “When I receive an email — doesn’t matter what it is — I thank the person for emailing. He paused to consider the situation and delayed his work longer to write us. That’s huge. That’s model behavior.”
Be visible.
From the employee’s vantage point, so much about security is initially unknown: a malicious hacker, the origin of a breach and the extent of the compromise. Because of this, it’s important that the security team is not invisible. “Being visible — and recognizable — to the team is one of the responsibilities that falls into security management. Not to reinforce stereotypes, but a lot of people are in security because they find problems more interesting than people. It’s not that they don’t care about people, they are just deeply fascinated by the technical,” says Graham. “They find deconstruction more interesting than building. They find breaking a system that's well created more interesting than creating a system well. You have a lot of people in security who are into figuring out how someone else is breaking something and doing it on a very deep, technical level that takes extreme focus.”
That intense study is not always conducive to being socially engaged at work, but security leadership must be. “Especially after a startup really grows its headcount, it’s important to take easy opportunities to be visible in the organization. How many employees can pick out one of their security people out of a line-up of colleagues? Security needs to be more public to put a face to their work. Employees are more likely to reach out to a name or face that they can recognize it,” says Graham. “Otherwise, if you’re absent or unfriendly, the ‘big brother’ or ‘babysitter’ attribute will creep back into security’s reputation. At Evernote, Zynga and Box, security leaders were social, present and recognizable internally. That demeanor is probably the hardest personal growth milestone to have a career as a security leader. The best I know are engaging, encourage interaction, and are earnest about it.”
The Future of Security
With nearly two decades in security under his belt, Graham has a unique grasp of history when it comes to security. Looking to the future, he sees the field maturing as it draws practices from other industries. “In the last couple of years, security has more heavily borrowed from statistics and the insurance industry. Everyone has heard the refrain: ‘You can’t predict what hackers are going to do.’ It’s false,” says Graham. “Sure, I can’t anticipate what one hacker will do on one system through one vulnerability, but we can absolutely track what hackers as a collective will do and begin to predict the types of breaches that result.”
Graham draws a parallel to kidnapping insurance. “Every kidnapping is a unique event with specific humans deciding to go after an individual in a particular way. Yet, despite being a distinct event, I can get insurance for it. What that means is that the insurance industry has figured out the mechanics and economics behind the big picture,” he says, “They’ve figured out the averages and know that if they insure types of people based on certain classification decisions, they can make a profit over the lifetime of the policy.”
There have also been recent advances with statistical analysis in the compliance world. “There are very big companies who act like they know how to do it, but the truth is that this is an emerging field. What’s happened recently is that statisticians have worked with security teams to test practices that actually lead to useful outputs,” says Graham. “I think in under a decade, we’ll be able to start making long-term predictions about security — in ways that are meaningful to the business. That’s part of what I’m working on now at Box. Before long, security teams everywhere will be able to talk to the business not about risk as a set of abstract threats, but in terms of losses — or saved gains — in dollar figures.”
We, in security, are on the verge of accurately quantifying risks and breaches in dollar figures. It’ll make dialogue and decision-making with leadership so much better.
In a founder’s fast-moving quest for engineers, product-market fit and profitability, it’s easy to see how security might fall from the priority list. That is why Graham outlines simple, highly leveraged adjustments that have a big impact. For starters, take small security steps from the beginning, but bring on your first full-time security hire after you’ve hit 30 employees. Set up security for greater results by training employees to take pause with trust decisions, killing the shame game and removing any friction to proactive employee outreach. Security leaders, put a face to a name and your function.
“Protecting people and property is an age-old service. However, when those people are dynamic digital natives and you’re talking intellectual property, it’s a different — and still developing — game. There are mature stable frameworks and languages that can solve half of the security problems out there at the onset, but no one uses them because they’re uninteresting,” says Graham. “In many ways, the field is still open frontier. Security is an emergent meta-practice, working with and around the constantly evolving fields that make up the tech industry. As long as those core practices remain fluid, security will continue to be as much of an art as it is a science. As we work to make it more statistics-based, security will shift to be a more rigorous, engineering-based practice. But in the meantime, if you aren’t sure, just email the security team.”