Should Your Data Science Team be Centralized or Decentralized? Why Not Intentionally Rotate?
Yuri Balasanov has seen another interesting approach.
Our recent Managerial Analytics class discussed whether data science teams should be centralized or decentralized.
Our discussion used material from Marta Magnuszewska’s talk at Northwestern’s 2023 Analytics Exchange. Marta is currently with Markel Insurance and has built multiple data science teams.
She gave a good overview of the pros and cons of centralized versus decentralized. A centralized team has a stronger community and can leverage knowledge. However, it takes longer to find and understand the business problems since it is central. A decentralized team is more customer-focused but suffers from duplication and lack of collaboration. At Opex Analytics, we were decentralized because we worked directly with client teams. As a result, we noticed many of the same engine types were being built multiple times.
Marta made a case that a hybrid approach might be a good compromise. I’ve always worried that it is easy for a hybrid approach to be the worst of both.
Yuri then gave an example he’s been seeing. Some companies are intentionally switching from decentralized to centralized every couple of years. This idea is interesting and inspired by the famous multi-armed bandit problem1.
The basic idea is this: the decentralized teams explore many different types of solutions and are given the freedom to be creative. Of course, there will be a lot of duplication, but that comes with the territory.
Then, the team is reorganized to be centralized. The centralized team then doubles down and scales the best ideas from the decentralized teams.
At some point, the centralized team has reaped the value of scale, and the new ideas slow down. Then, the process is repeated.
The obvious downside to this approach is reorganizing is never easy, even if intentional and planned.
I’m not an expert in organizational design. However, I’ve been around long enough to know that this debate will rightfully continue. The idea of an intentional switch should at least be added to the discussion.
If multiple slot machines have different (but unknown) payout odds, you want to explore each slot machine several times before picking the one with the best odds and just playing that one (called exploiting). This is a probability problem. It shows up in many real-world situations where you need to explore the solution space before picking a direction.