Exago Logo
Generic filters
Exact matches only

I’ve been involved with hiring at Exago, Inc. for about three years. In that time, I’ve learned that hiring technical people at a software company is really hard and demands long-term commitment in order to produce sustained success.

This is the interview process we created and then iterated on at Exago (let’s exchange notes about your process). The title of this article references The Hard Thing About Hard Things by Ben Horowitz.

The Phone Screen

Our hiring process has multiple phases and resembles the marketing funnel. A relatively large number of candidates exist at the top of the funnel, and we want to spend the least amount of time possible screening out those who are obviously unqualified. As candidates move “down” in the funnel, we have progressively higher confidence in their fit for the role and invest an increasing amount of time in them.

After being prequalified by our hiring guru, Jeff Kalpakis, who oversees new hire recruitment and staff development, and submitting a small code sample through third-party software, the top of the funnel feeds candidates into a technical phone screen.

Our technical phone screen is a 20-minute call designed as a low-cost way to quickly discover if the candidate is a poor fit. I ask five technical questions to test for a “vacuum” of knowledge in any of five basic areas: recursion, time/space complexity analysis, object-oriented programming, data structures, and scripting. The questions are the same for every candidate, so I am able to measure responses against a stationary goalpost. Some great candidates don’t always know specific academic terminology, and that’s not a disqualifier. We look for a lack of conceptual understanding of any of the categories.

The Career History Interview and Tech Quiz

After candidates are qualified through the phone screen, they are invited into the office for a sit-down interview and technical assessment. The interview serves to educate us (I always interview in pairs with another engineer) on the candidate’s skills and experience. This largely borrows techniques from the topgrading method. This interview is our chance to dive deep and uncover potential red flags in the candidate’s history, ask for evidence of past success, and evaluate the candidate’s technical communication skills.

I’ve found the best developers want to talk about their past projects in too much detail and need to be interrupted to keep the interview moving. This is a sign of a developer being truly passionate about their art versus someone who simply regurgitates their résumé to you in person.

After the career history interview, the candidate is invited to complete a technical quiz at one of our workstations. The quiz has three questions and tests to see if the candidate can actually write code. This usually takes 45 minutes to an hour and may be completed in the candidate’s language of choice. Sometimes candidates fumble on one or more parts of the quiz, and this allows us to evaluate how quickly they are able to take direction when working on code.

The “Final Round”

Having passed the first in-office interview, the candidate is invited back to the office with the following agenda:

  1. Architect a mock web application on the whiteboard.
  2. Work on an actual incomplete implementation of the application and fix bugs.
  3. Interview with people from other departments.
  4. Have a sit-down talk with our CTO.

Each of the above takes about 45 minutes, and we may end the interview early if we decide the candidate is not a good fit in steps one or two. We call the first two steps our “workday” test. This practical testing environment has proven to be a great way for us to get to know what it might be like actually working with a given candidate. It turns out that some candidates present very well on academically focused tests and interviews, but end up having a very difficult time doing actual design and coding in the workday test.

Building a Framework of Patches

Our interviews don’t last for months at a time, and there is no way to be absolutely sure a candidate will be a good fit for the role. Sometimes we make hiring mistakes and bring on a candidate who is quickly discovered to be a poor fit. I treat such mistakes as “holes” in our interview process that can be patched.

After identifying that you have hired a person who doesn’t fit the role, it’s important to do root-cause analysis, find why that lack of fit wasn’t discovered in the first place, then patch the interview process to screen for that case in the future. This is how our “final round” interview phase, with team-based white-boarding and coding segments, came to be. We were struggling with the highly speculative nature of hiring a candidate after only completing a sit-down interview and brief tech quiz, and wanted a way to know how the candidate would be to work with before making a job offer. The best way to know how someone is to work with is to actually work with them!

It’s also critical to make sure to have job responsibilities explicitly written down before hiring so you actually know what skills you are hiring for. Job criteria should be expressed in terms of outcomes rather than qualities. Listing “Communicates sprint progress and warning signs every Monday to the Development Manager” on your rubric creates a clear and measurable expected outcome. On the other hand, list “Great communication skills!” and you will find an endless supply of “great communicators” with no way of measuring their performance in that dimension.

Outstanding Questions

We treat our hiring process like an engineering problem and constantly question existing methods. Some open issues we are currently discussing include:

  • Our company is not located in a major tech hub, and we often feel we have a low volume of strong applicants. Should we consider remote positions on a wider basis?
  • Is our testing too academically focused or tool/ language specific?
  • Should we start sending candidates take-home assignments?
  • How can we be less myopic and focus on the long game — i.e., cultivating relationships and generating awareness/interest early to drive more qualified leads to the top of the funnel?

The Bottom Line

So back to the title of the article: The hard thing about hiring engineers is … well … it’s hard! Vetting people on their professional merits is an unnatural and sometimes uncomfortable process. Hiring managers (at least good ones) usually means you never feel 100 percent secure in any hiring decision. But to be less depressing and vague, I have learned some more tangible points as a hiring manager:

  • References beat all — friends or former colleagues of current staff still have to go through the funnel but may skip early steps.
  • Every once in a while a candidate comes along who would be a clear win for the company. You need to be able to rely on culture, work environment, and interesting problem domains to bring the candidate across the finish line.
  • You can’t fake being passionate about extremely technical domains, and you can’t fake writing code. Both these things can be tested for.

At the end of the day, we strive to take every interview as a learning opportunity and optimize the process for our desired results over time. What are some ways you have improved the process of hiring for technical roles at your company?

Those interested in working at Exago may view our available positions on our Careers page. We’re always looking for self-motivated individuals to join our team.

Originally published by Software Executive Magazine.

BI Newsletter

Stay up-to-date on all things SaaS and analytics with fresh content each month.

Share This