Mar 13
Six Lessons Learned From Hyper-Growing a Data-Science Group
Some counter-intuitive and highly-effective insights I learned while establishing a group of ten Data Scientists and several models in production within a single year
When I joined Lightricks at the end of 2021, I was supposed to lead a small, somewhat experimental, team of 2 researchers plus me. That was perfect for me, as this would be my first official role as a manager. But on my first day on the job, the company’s business strategy changed — and my small experimental team was now in the very center of the company’s new focus, and it was pretty clear we are not going to be neither small nor experimental. And above all, it was clear we needed to move fast.
The true-magic we’ve performed over the past year is best described using numbers; on December 2021 we had a single product to support, no data to learn on and no infrastructure to run our non-existing models. By December 2022, we supported two completely different products, and had no less than six proprietary Deep-Learning-based models developed by us running in production, on infrastructure we set for ourselves, breaking every baseline and company expectations with uplifts of up to two-digits percents.
But that’s only half the story and the interesting numbers. I order to support all this, my team grew from the trio we were on our experimental team to no less than ten, transitioning from a single team to a group of teams, consisting of two Data Science teams and an MLOps squad — all within six months. Learning how to lead and manage this hyper-growing group was a whole other challenge, but this one was only my own. I was given the unplanned-yet-exciting privilege of designing a new group’s culture, and fit it to how I believe it would work best — and I’d like to share some of the most interesting insights I’ve realized during my group’s rapid evolution.
Deploy Fast, Then Rebuild
Think of this as a gentle variation of Mark Zuckerberg’s famous quote “move fast and break things”. As a brand new group, we had to deliver products fast — products no one ever used and no one could even asses their impact. Yet, none of these could be asses if we wouldn’t deliver something. And so, our approach to any new feature or product remains constantly the same — deliver something impactful fast, let the product establish itself and launch, and then we’ll use the feedback and time we get to rebuild it into a cutting-edge product. That assures we’re never blocking any product with extensive research, while also assisting us in getting the time, data and feedback we need to develop something extraordinary.
Outsource What You Can, Focus on What Matters
As I mentioned earlier, when the team first established, we had to develop not just our models from scratch, but also our infrastructure. Doing everything at once would be too much, and so I decided to outsource most of our infrastructure, and work with a new MLOps start-up which promised to save us months of work (and they did). They way I saw it, my infrastructure at that point of time was not my goal, but a tool. My goal was our models, and this is what I needed my team-members to focus on.
Outsourcing is not a decision which needs to be taken lightly. Besides the additional costs, there’s a concern of “stickiness” — the inability to cease future collaboration with a vendor. But on the other hand, developing your own tools also comes with a price-tag — employees salaries — as well as a time-tag, as the release of new products and models depend on tools which were yet to be developed. As stated previously, as we couldn’t really asses the true value of our new models and products, moving fast towards production and verifying our beliefs with the real world were our core values, and outsourcing was a crucial decision that led us in that direction. We launched our very first recommendation engine to production within less than six months of the team’s establishment, and we could not have done that if we would have done everything by ourselves.
I Should Not Be The Best
One of the most counter-intuitive realizations I ever received was from a great friend of mine, Marina Gandlin, who told me she realized that as a Data-Science team-lead, she shouldn’t be the best data-scientist in her team. She explained that if she will be the best, all tasks should be done — or at least reviewed and approved — by her, and she’ll quickly become a bottleneck. While approving everything might indeed be the manager responsibility in a really small team, when I got to the point my team had ten people and three different fields to work on, it was simply impossible. I had to delegate some of these responsibilities to what I defined at the time as Project Leads, who quickly became official Team Leads. And that means they needed to be the best, and certainly did their team-members. My job became to be the best in leading the best, and in many cases — that means to simply listen to what they have to say.
Overlapping Responsibilities
Probably one of the most counter-intuitive realizations we had — and the one of the most productive ones — is overlapping responsibilities.
Every once in a while someone is missing. They can be sick, on vacation or simply unavailable. But in small- (and even medium-) sized teams, each team member is the de-facto owner of a process, feature or a product. And when that owner is missing, open tasks related to their owned product are hindered, backlogged till their back. To tackle this, we came up with a new definition: sub-owner. Sub-owners are defined per feature and process by the team-leaders, so they know what they sub-own. Sub-owners must be one of the reviewers of every code-review and pull-request of their sub-owned product, and they also must approve it. The point here is to make sure sub-owners know their sub-owned products good enough to handle production issues and minor feature requests by themselves, without the owner’s supervision. This allows us to move fast, as we never rely on one single person.
Backwards-Mapping
If I need to point to the one question — or two, actually — that echoed through my mind from day one on the job, they’ll be How do I keep my team fully motivated and performing their best? and How do I make sure they don’t want to leave? I had all sorts of idea, but it all felt like a chaos of different directions and I had no idea where to start.
So I started with the end. I tried to describe what my ideal group should look like from a very high-level point of view — things like “employees are happy”, “employees realize their full potential”, etc. From there, I started asking what would assist in achieving these high-level goals — things like “reducing stress”, “reducing friction”, “access to experts”, etc. Then again, I broke down these goals into actionable items I can actually perform — things like “respecting off-hours”, “async updates”, “effective onboarding”, “paid experts consulting”, etc. I ended up with a map leading directly to my ultimate goal of the ideal group:
This backwards-map became my go-to document whenever I had a new initiative in mind. I would open the map, see where does this new initiative fits, does it support one of my high-level goals, and if there are any other things I can do as well. Many of my insights below were initially derived from this map (which probably deserves its own blogpost, but that’s for another time).
Work Comes Second
One of the things I quickly realized was that if I want to keep my employees with me for the long run (and I do), I need to focus on two things — they’re happy where they are, and they’re not worn out. Not wearing employees out doesn’t mean we schedule coffee breaks and team lunches, it means I fully understand their priorities in life — Family, friends and themselves come first, work comes second. If I’ll respect that and fully embrace this, I’ll have my employees onboard with me because they want to, not because they need to.
Accepting being second-in-line is more than just reminding employees to take time off. I make sure that I do not interrupt them on their off-time, unless it’s an emergency. If I want to send an email or a Slack message during off-time, I schedule it to the next morning. I expect them to be completely logged-out of work communication when they’re on vacation. I personally urged my male employees to join or replace their wives on maternity leave for as long as they want (and I personally did it myself, too). The result are employees who don’t need to seek balance — it’s provided to them by default. And such employees are the most engaged ones you’ll ever get.
It’s hard, and even strangely weird, summarizing an entire year in a single post — but I hope this will be of value to some of you! And if you have any unique insight or realization you can share with me too, please let me know :-)
