Open Ecosystem at Samsung

In the past 5 years, Samsung has invested a lot of effort in creating the team, culture, and processes to become an active contributor to the open ecosystem. Why would a hardware company invest in this? What is the value? How do we marry hardware differentiation with open source and standards?

In this blog post, we try to clarify this. Specifically, we will detail (i) what the Open Ecosystem is for Samsung, (ii) why we believe that openness through standards and open source are important, and (iii) how Hardware & Ecosystem Co-Design is key for being successful as a hardware vendor. Finally, we will (iv) go through the history of creating an open ecosystem team across Samsung and the lessons learned in the process.

1. Open Ecosystem in a Hardware Company?

Before we explain the role of the open ecosystem, we will spend some time defining it. At Samsung, we define the ‘Open Ecosystem’ (usually simply referred to internally as eco) as a combination of Standards, Open Source, and Industry Synergy. A customer solution is then the sum of hardware and eco.

• Open Source. For us, open source refers strictly to widely used, mainline open source projects where the contributions make it upstream. We do not believe that forking projects and adding features on top serves the purpose of contributing to the open ecosystem. It can be a way to develop new features in the open (and we do this!), but the ultimate goal is always to push the code upstream and merge mainline so that it is available through enterprise distributions. In the few cases where we see a hole in the ecosystem, we do start new projects, which we develop and maintain to support different customer needs, while maintaining a healthy community around them. One such example is xNVMe.

• Standards. We participate in a wide range of standard bodies that are relevant for our products. We contribute and participate on the leadership team of some of them.

• Industry Synergy. One of the challenges of working in open source and standards, is that a common infrastructure should be able to cover several use cases. There are three sides to this problem: (i) different companies might have a common use-case, but internally, they devise different way of approaching it; (ii) a new feature supported by a standard might be relevant to one or several companies, but the changes required to an open source project might incur in major changes; (iii) different companies manage to standardize (in the same or different standard bodies) similar (but not identical) approaches to solve the same use case. In all these cases, there is a need for the industry at large to take trade-offs and make compromises so that we can cover as many use cases as possible, while making projects and standards easy to maintain.

By looking at eco as a combination of these 3 components, we make the upfront decision to see new product features as a combination of a community effort and an internal drive to provide the best hardware solution. When we propose or support new features, we work with peers, competitors and customers in open conferences, boards, and standards bodies to take the right trade-offs so that the final feature can support as many use cases as possible. We then make a commitment to support customers in mainline open source projects, where we push upstream the necessary changes and work with the community to make the code useful and maintainable.

When a new technology arrives, a few pioneers build vertical solutions that tightly integrate hardware and software, providing new features as a differentiating factor at the cost of locking consumers into a single source. However, as the technology matures, the existence of an open ecosystem around it is needed for it to become widely adopted. Let's take NVMe as an example. The earliest versions of NAND on PCIe relied on vendor-specific protocols and dedicated software for operating the hardware. A first mover in this space was Fusion I/O, who produced the PCIe-attached SSDs and the host-based Flash Translation Layer (FTL) on the side. The performance of this solution was unprecedented, as the generally available SSDs communicated through SCSI. As more vendors entered the NAND on PCIe market, the need for a standard arose and the NVM Express (NVMe) Consortium was created. Today, NVMe is the main protocol for PCIe-attached non-volatile media, and the open ecosystem around it ensures that vendors and consumers can rely on a fast, secure, and stable software stack that is able to support the latest additions to the specification. We see this happening again and again across different segments of the industry. We identify 3 main reasons for this:

• Supply chain & Multi-vendor. When a new differentiating technology appears, the benefits of being the first mover surpasses the risk of relying on a single vendor. As more vendors enter the market, the differentiating factor dilutes, and the risk of being single-source is typically not worth it. The problem is: how do vendors align around a protocol, consumer behavioral expectations, and at the same time manage their own risk of jumping into a completely new technology? The answer is standards, open source, and an open community for the industry to understand common use cases. Apart from distributing the risk, working in the open completely eliminates lack of trust issues. This open ecosystem provides a stable multi-vendor supply chain for consumers and reduces the hardware fragmentation for vendors, effectively reducing the amount of Stock Keeping Units (SKUs) that need to be validated and qualified.

• Software stability, security, and neutrality. Relying on mainline open source projects to support standards comes with the intrinsic software quality associated with it. Projects like the linux kernel implement strict code-review processes before merging new patches. Also, several companies and enterprise linux vendors run continuous integration on it to prevent regressions. Moreover, mainline projects are normally maintained by individuals that for the most part know how to wear neutral hats when it comes to adding features and maintain a stable codebase. Since conversations are always held in an open forum, any signs of nepotism favoring a particular company are very rare and normally addressed within the specific open source community. This allows both vendors and consumers to send patches to support new additions to standards. Note that here we refer to core, infrastructure open source projects (e.g., Linux kernel, core libraries, toolchains), which provide the base for hardware support. On top of this, consumers normally then add their differentiating software and services, which are the value-add for their own customers.

• Shared risk for new technology adoption.  Adding new features to hardware products is a multi-year effort with significant investment from the vendor side. Supporting different customers with different requirements adds complexity and fragmentation, thus making this investment unsustainable. Standards help address this issue. While adding features to standards is also a multi-year project, the fact that vendors and consumers need to agree in an open environment and take trade-offs reduces errors and fragmentation, and makes it possible for a feature to target different use cases at once. This is only amplified when these features are implemented in mainline open source projects, where different companies collaborate to accelerate feature adoption. By the time the hardware is available, the ecosystem is available. This spreads the risk across companies and makes it possible for more complex features to be successfully deployed.

2. HW & Ecosystem Co-Design

Hardware and open ecosystem co-design is a natural evolution from this mindset. It involves considering the ecosystem around a hardware feature at conception time and defining how the hardware will be supported as part of the business strategy. In order to get to this point, we see vendors having to go through 2 non-parallelizable phases.

The first phase is drawing the line from HW to eco. This is thinking of the open ecosystem as a vehicle to support hardware features. Most hardware companies start their journey in open source and standards this way: they upstream drivers for their SoC, they enable existing features in the drivers managing their hardware, or they contribute newly developed standard features to core subsystems. Most hardware companies understand the value of doing this and in one way or another aim at being part of the open community that enables hardware in their domain. Companies doing this understand that the path to mass adoption goes through upstream linux.
Stopping here brings a couple of challenges:

• Feature Echo-Chamber. When vendors look at the world through a HW to eco lens, all they see is how they can add new features and differentiate their products through their HW architecture. Typically, these advancements come as a combination of internal teams proposing optimizations and customers proposing new use cases. Here, the hardware features are designed and implemented. In the worst cases, the hardware tapes out before anyone in the company has looked at how this feature will integrate in the ecosystem. You need to be very lucky for this process to end in a standard and in any mainline project. Often times, this creates SKU fragmentation as the hardware only targets specific customers and requires a dedicated software stack to support it.

• Industry Misalignment. When a new technology becomes a major point of differentiation, some companies tend to become blind to the feedback of the industry and the open communities. This can be caused because of a misunderstanding of the use case, an exacerbation of the real value of the technology, or by underestimating the scope of the changes in a particular standard or open source project when it comes to supporting this feature. Misaligning with the industry is risky as it puts pressure in the community to support a technology that ends up not being used and keeps taking maintainership efforts as projects evolve.

• Lack of Trust. Ultimately, all the above results in the company losing the trust of the community as it is seen as an enterprise that only cares about pushing its own features without considering the effects in the rest of the community. This is especially true in mainline open source projects, where maintaining a healthy codebase is more important than supporting the latest protocol features.

The second phase is drawing the line from eco to HW. This is defining hardware requirements through the understanding of use cases in the open ecosystem that can benefit for better hardware and software interaction. Here, requirements are brought into the hardware after observing a problem on the ecosystem. The key point here is that it might not be an issue that has been explicitly identified by a customer (and thus requested), but rather a well-understood and established nuisance that would eventually need to be addressed. Thus, there is a need for a strong open ecosystem understanding from within the company, as new features will be implemented in hardware based on this. Moreover, once the ecosystem is part of the product strategy, it is natural to contribute to the well-being of the standards or open source project, even when this benefits everyone as it is understood that they are a key factor for business success.

3. Global Open-ecoSystem Team (GOST)

The remaining question is: How can a large company drive such a significant change in their culture and world view? At Samsung, we built a dedicated group to drive this effort. We call it GOST: Samsung's Global Open-ecoSystem Team. Originally, GOST stood for Global Open-Source Team; as the scope increased to more ecosystem activities, we adapted.

The GOST is a globally distributed team that focuses on different aspects of the ecosystem.

 
We contribute to a wide range of open source projects including the Linux Kernel, QEMU, and SPDK. Here, we do not only contribute with code and with maintainership responsibilities, but we also put effort in helping drive the community by being part of several open source organization boards, leadership committees, and conference organizational bodies.

Beyond ensuring the high quality of the code, one of our priorities is also the well-being of the people behind that code. The overall open source community is primarily a great place to work, and while we are sometimes in the spot for heated conversations, we see this as a consequence of two things: (i) the pressure that our maintainers carry, dealing with hundreds of daily patches and ensuring that projects are driven forward; and (ii) the fact that we carry all conversations (also difficult ones) in the open. Given this context, we do spend time on mentoring new members (inside and outside of Samsung) in how to get started on contributing and dealing with harsh code reviews.

As a highly distributed team, we have presence in Korea, USA, India, China, Israel, and Europe (through Denmark), which follows some of Samsung's largest development hubs across the globe. We believe this is key to the success of our ambitious ecosystem vision because of 3 main reasons:

• Focus on Alignment and Role & Responsibility. Leading a distributed team requires makes it explicit that the focus should be on alignment and not on task management. This helps develop a good culture, where leaders provide clarity on what is important from the business and product perspective, and the different ecosystem teams can drive the plan and execution. Since teams within the same project are likely distributed, it is natural to spend time in defining roles and responsibilities (RnR). Overall, our strategy at Samsung has been to follow the path of constantly realigning the GOST activities with our business strategy and then provide clarity to the team, which then becomes part of our tactics and execution. 

• Need for Locality. While distributed engineering teams can operate efficiently, it is a challenge to drive a fully distributed vertical strategy across planning, ecosystem, and product (hardware + firmware). This is especially true in places where the ecosystem team is the only distributed team (which is a relatively common setup). Other factors such as the benefits of face time across the team, white-boarding, or simply human interaction play a big role when driving new ambitious projects. At Samsung, we designed the GOST to be distributed form the beginning. Most members are close to a development hub, but we do have fully remote colleagues. We believe that maintaining a distributed team that is able to interact with local teams that are related to them is crucial for long-term success.

• Access to Global Talent. This is the old "hire the talent where the talent is". The open ecosystem community pioneered remote working years ago, and since this is the pool of people that we target for the GOST, it is only natural that we follow this philosophy.

4. What's next? Any joiners?

For us, the next step is to continue doing what we are doing, while growing the team to different areas. We are fortunate that Samsung is a large company with many business units that can benefit from understanding and contributing to the open ecosystem.

As for next steps externally, we hope to see more hardware vendors be active in standards and open source. It is quite often that we meet industry colleagues at conferences that praise the work or report upstream issues for work that we and others have done. While we are happy to see our community efforts recognized and enjoy fixing issues reported by the community, we would hope that their companies decide to participate actively and bring the conversation to an open forum. This would help many more technologies to be successful by mitigating the echo-chamber and industry misalignment issues discussed earlier in this post. If nothing else, we hope that this post provides a blueprint for how to get started.

Finally, this blog post is the first one in a new blog initiative within GOST. We plan to post about technology that we deem relevant, with a focus on the open ecosystem. We plan for in-depth posts at a 6 to 8 week cadence. If you think this is interesting, stay tuned!