At-Launch Linux Hardware Enablement

The Linux market presents some unique challenges to Independent Hardware Vendors (IHVs) in bringing their products to market with broad support available at the time of launch.  Independent of ideological or pragmatic rationale, both Open Source and proprietary drivers are constrained by similar mechanics.  This article provides a broad outline of the mechanics and considerations that are needed for delivering hardware support at-launch.

Aim of Enablement

The fundamental aim of enablement is to allow the majority of important features to be available at the time of launch. This enables Linux usage of hardware as OEM programs approach launch and the channel begins to fill with the hardware.  Under Windows, the enablement is typically a direct byproduct of the development cycle.  Drivers are developed and WHQLed, and finally included on the CD shipped with hardware or posted on the internet.  Linux presents some unique challenges which are not impossible to overcome but may require considerable advance planning and some tough corporate decisions.

The Generic Open Source Process

Of course, historically, a lot of HW component manufacturers have either ignored the Linux market or have enabled community developers to do some or all of the work for them.  The post-launch availability of hardware to developers either through direct seeding by the IHV or through general market availability  doesn’t guarantee end-user experience in any way against any timeframe.

For At-Launch enablement IHVs have to do all the work prior to the release.

Critical User Scenarios

Linux deployments are notoriously difficult to quantify, however ignoring that segment can have materially negative impact to sales or bad publicity. The following are a core set of user and associated scenarios that have been considered within this article. The list is by no means exhaustive and the reader may need to extend these scenarios to suit a particular focus, intent or market segment.  The users and scenarios are intended to drive a background for the subsequent sections of this article.

Launch Reviewers

These users receive the hardware as part of the seeding process for reviews of the product. The reviews are usually intended to highlight particular benefits of the product, however as expected, the reviewer may be sensitized to other areas of the product that do not have as good a story. Due to the fluid nature of media on the Internet, Linux has a higher and louder voice than through mainstream technologies in the traditional media

Channel System Integrators

These users take the products in the channel and build systems as part of a commercial enterprise. For some IHVs, this represents a critical part of their business. However, as indicated earlier, quantification of this portion of the market is difficult. The channel is typically aggressive, whereby a fully working product is needed as hardware becomes available.  Some IHVs have a clear Linux taint to their channel, knowing that a material portion of the revenue the IHV receives from the channel comes from Linux.

Channel End Users

These are the recipient of either the system integrator market or the end-user purchasing hardware at beige-box manufacturers or retailers. These users are typically technically savvy and would have a higher likelihood of having a partial or heavy interest in running the hardware with Linux.

OEM

OEMs face a more difficult problem. To constrain support costs, most OEMs will limit the formal support of Linux, however their support infrastructure will have continual requests for Linux support.  A lot of times, they may just be looking for “good enough” or “doesn’t crash the system”.

On OEM platforms that do require Linux support for sale, the OEM will typically interact directly with the IHV, requiring a complete – or near complete solution for the “Gold Master” image that is deployed in the factory and delivered to the end-user.  This is similar to the Windows model that supports the IHVs ecosystem very well.  Unfortunately, there is a tendency to not gate the release of hardware due to tardy Linux support, but to follow-up days, weeks or months later with the support for Linux users.

OEM End Users

Similar to the Channel End Users, there is a subset that proactively use Linux. Unless the OEM has a dedicated SKU targeted for deployment with Linux, most users with look to the IHV of the components in the system for support. This is a byproduct of the historic driver focus that the Linux market has evolved with over the last two decades.

As an aside, this doesn’t “follow the money” of the value chain and is unlikely to cause a large impact to the decisions and driving criteria for the IHV.  The diagram below highlights the money train or value chain, the dotted line shows the typical end-user channel.  As the old adage goes “money talks” and so the channel to the IHV is through the OEM.

Barriers to Enablement

The Linux market represents a unique set of challenges. Historically, most hardware would not have Linux drivers, and so the Linux market would receive drivers at six to twelve months after release.  That is assuming that there are sufficiently capable developers available and the IHV hasn’t made their product closed to external drivers through some technical feature.

As the market has matured, the expectations of support from the IHV have increased and the window of time acceptable by the vocal community has decreased.

The following are some of the barriers that affect the ability for an IHV to have at-launch enablement.

Inherent Open Source Hysteresis

Depending on the particular component and particular sub-project that requires support for new hardware, there is a delay between when the code is pushed upstream and when it can be considered generally available. As a broad generalization, the typical flow of a piece of code will be as follows.

Note that some projects may have more steps which only further delay the upstream to general availability window. This upstream push can vary between three to six months up to a number of years. This impact not only affects Open Source drivers, but also proprietary drivers since they will carry some interaction with the software ecosystem that they enable.

Obviously there are a number of short cuts that can be put in place by the IHV, user, distribution or upstream project.  Each shortcut that is put in place will usually carry an integration cost as changes are back-ported to older APIs, users need to use different kernels, etc.  Companies such as Red Hat already accept this as a cost within the market and expend large engineering resources on bringing in new hardware that is required by their supported customers onto distributions that are using old kernels.  The costs of doing so increase the longer the current upstream software is given time to evolve.

Tyranny of the Upstream

The majority of new development will occur upstream within components that are also undergoing development. A complex, but critical example is for X drivers. Assuming an Open Source graphics driver, the DRM, DDX and OpenGL versions are all independently developed. The most efficient way to develop an individual component is to base it off recent releases or even the upstream tip-of-tree.

This has a side-effect of introducing dependencies between components. A user may want to get fast 2D acceleration, but may ultimately end up needing to download, configure and build a large number of components to achieve what was originally considered a simple update. This is ignoring the technical complexity that exists in building some of the larger Open Source components. The majority of users in the above categories will not have the expertise needed to build the components, or be willing to invest the non-trivial amount of time it takes to understand the method of building some Open Source components.

Traditional Operating Systems like Windows or OS X increment driver APIs at a considerably slower rate, making operating system alignment something which is explicitly planned for and executed usually in the context of OEMs shipping an IHVs product. The driver architecture changes rarely within a product cycle. Within Open Source, the ABI or API may change in non-trivial ways on a week-by-week basis.

Again, proprietary drivers are affected by this too, there may be ecosystem architectural changes that may need to be supported and these may have nontrivial impact across even a proprietary driver.

A final consideration regarding upstream is there are problems with downstream usage of a driver, there is a likelihood that the upstream will be fixed.  With some expectations, most of the upstream developers exist happily in their world of “tips of tree” and “just committed”.  Unfortunately the end users may be forced to either back-port the code themselves, or go through a learning experience of finding  the ecosystem dependencies for their driver and how that differs from the ecosystem already installed on the system as part of the distribution.  Many Linux installations have had to be re-installed when a supposed benign dependency from upstream clashes horribly with other software already installed.

Market Signalling

Particularly in some IHV market segments, there is are competitive reasons to not signal to the market features of upcoming products. The hysteresis identified above indicates the advance timeframe that an IHV needs to plan for with a pure Open Source play. Messaging to the other IHVs in the market that there is an increment in CPU cores, or a new graphics technology is becoming available may permit competitors to make concrete changes to remove some competitive advantage.

For hardware that is required for a system to boot, such as CPUs, this model is reasonably well-managed.  The value of having a new CPU boot  far outweighs the cost or risk of signalling a newer CPU core is on its way.  This is particularly important early adopter markets for CPUs like server.  The IHV will generally work with both the upstream and the downstream distributions to ensure that at launch the majority of relevant distributions will work out of the box.

A generally accepted approach to this model is to use some sort of embargoed changes that will be pushed upstream, but are already integrated into particular distributions kernels and driver sets.  This does place some contention and pushes some of the risk to the distribution vendor, since if the upstream requires heavy changes before accepting the drivers, the distribution will effectively have to double the work to enable the IHV.

Resource Sizing and Delivery

There are a number of complex issues driving the size of the resource pool needed to have consistent support at the time of release. The tempo of hardware release, the potential for multiple code streams for compatibility, ecosystem changes around the drivers, support of users, etc all contribute to the cost of building a driver considerably more than “Just a bunch of developers”.

Likewise, assuming that simply creating an Open Source driver will allow a large number of developers to assist in development is not something that is guaranteed, and should not be considered critical for an IHV’s driver.  The only resources that you can rely on are the ones that you pay.  Be it via in-house staff or by farming the work out to a suitable third-party.

Solutions

Communication

The essence of hardware enablement is to ensure that there are drivers available that support the primary features of a product at launch. So long as the mechanism for the hardware enablement is clear, unambiguous, simple and reliable, different options are available. Unfortunately, the messaging around Linux support from IHVs is typically hidden, ambiguous and complex and unreliable weeks after authoring.  This is a challenge to the IHV who typically will not have the PR, outreach and other channels pushing Linux awareness the same way as windows.

Pareto Principle

Although there is a very fragmented distribution market, there are a few clear leaders in both commercial and community distributions. Enabling a small percentage of those distributions will typically enable the majority of users. Depending on the market, Red Hat, openSuSE or Ubuntu may represent the lions-share of relevant deployments. Likewise releases within those distributions will have a small number of versions of interest for large deployments.

By carefully choosing which distributions and will get enabled an IHV can make or break its launch time activities.

Real Driver Scenarios

Again, not exhaustive, the following represent the primary options that are available to an IHV for supporting their product. A brief discussions is included in each as to how to overcome some of the critical issues raised previously.

Pure Open Source

Drivers are all upstream at point of release. Depending on the sensitivity of the product being released, the “push to upstream” may be at the point of release or may be many months earlier. The hysteresis mentioned previously is typically overcome by investing with the selected distribution partners and ensuring that the hardware support is integrated into the distribution in the release or update cycle ahead of the release of the product.

Unless resourcing has been planned for specification release and external developer education, the a pure Open Source model provides minimal community involvement and consequent increases to the developer pool. Almost all the development and release will be continue to be done internally to the IHV.

Hybrid Open Source/Proprietary

Typical key IP will be captured in a subset of the device. A common architectural approach is to have an Open Source shell that provides basic functionality with a Proprietary extension which exposes the new features. Most IHVs who operate in this fashion tend to run into community push-back and reverse engineering, or end up being marginalized by the shortcuts needed in the driver to enable the hybrid approach.

Proprietary

Drivers are released from the IHV independent of the Open Source components that they rely on. This forces higher engineering efforts within the IHV, but arguably can provide a better experience for the OEM, system integrators and end users since the IHV controls the majority of the efforts around development and release. Sub-models within Proprietary also include options of a clean-room driver, or a driver that contains code that is common across multiple OS platforms.

Multiple Drivers

To provide a solid at-launch experience as well as to seed the market for the future, the requirement to do multiple levels of concurrent engineering cannot be escaped. Even with a full Open Source model, the support that may be necessary for laggard distributions may represent an effective re-write of the entire driver multiple times to support the business needs. A full proprietary model may or may not be possible depending on which IHV products need drivers and the ability to post-install a driver without issue.

Open Issues

Independent of the model of driver that is used, there are still a set of open issues that cannot be addressed outside the context of the IHV. The following serve as discussion points or starting points in thought exercises.

Full Feature Coverage

Modern products are sold based on the sum total of their features. The deciding line between availability for support some features may create an incumbent with the right mix of features. All features may not be necessary, but a core set of features need to be carefully considered and planned.

For example, NVidia’s support for OpenGL and VDPAU has allowed the Linux-based market for HTPC interfaces circle near exclusively on their product families. No other vendor can offer boast such a compelling 3D solution and video solution.

Embargo Risk

A typical mechanism for seeding an ecosystem is to have drivers or information under embargo until a certain date. Dependent on commercial relationships between the IHV and Distribution vendor, there may be opportunity to synchronize the release of drivers with a HW or distribution release.

As with any 3rd party release of information, there is the risk that certain competitively sensitive information may inadvertently go public, independent of the embargo date.

Certification

Linux has had certification as an available option for OEM and IHVs for most of the existence of the market. Unfortunately, a lot of the certification is more of a validation that absolute baseline functionality works (ie pixels appear on the screen). Certification generally does not necessarily entail any measure of usability for the user.

Skewed Business Rationalization

As mentioned previously, the market has skewed the industry fairly heavily. The direct and measurable commercial impact of Linux is typically very low. Consequently, the market does not make Linux a requirement to the same level as it does Windows, this heavily affects both the perception of Linux’s value and resulting investment within an IHV or OEM. Unfortunately, the noise that the community can create when an IHV makes a misstep is in-congruent with this quantifiable opportunity.

This becomes critical within the OEM and IHV space, where the OEM produces a program with explicit engineering requirements on the IHV, and yet once an OEM program is released, the IHV bears the brunt of stability or component issues relative to Linux. Since the OEM is not part of those customer interactions, the OEM doesn’t need to worry as much about Linux. Hence the OEM doesn’t place Linux as a requirement on the IHV, and consequently the IHV has to invest further into support for Linux for hardware that OEMs don’t require Linux support for and yet the Windows feature set is driven primarily by OEMs.

Distribution Fragmentation

The Linux community has always held in high esteem the low barriers to creating a new distribution or variant of a subsystem. Arguably, this fragmentation has allowed innovation to proceed at a lower cost and higher pace than the rest of the industry. However, for the IHV or even ISV, this represents a heavy cost in development and validation across multiple distributions and multiple releases. Across particular ABI fracture points the drivers may need to diverge by a non-trivial amount, further adding to costs of development.

ROI Quantification

How much to invest is a very difficult questions and is typically driven by the quality of the quantifiable “return”. If a product has a Linux niche, the ROI can be made easily as a subset of revenue for that product niche. If not, the return may not be monetary, but may come down to “we’ll invest enough to make the noise go away”, whereby the lack of negative press represents the quantified benefits.

Non-Engineering costs

Moving beyond technical and engineering concerns, an IHV also needs to determine how much effort it would be willing to invest into simplifying effort to enable their product. As with any customer interaction there are non-trivial costs required to maintain online resources and adding polish to the IHV driver-set. These need to be factored into the ROI quantification as well.

Passive vs Active Involvement

A further consideration is to what level the IHV wants to become part of the community. Developing close to upstream and partaking in the general efforts may represent a strategic benefit, but it is still a cost that diverts engineers away from core development on behalf of the IHV.

Tempo of hardware change vs Developer Resources

Depending on the industry that an IHV is operating within, the tempo of the release of the hardware will drive the appropriate solution.

For example, if a product has a typical release tempo of 24 months, and a typical full featured driver development effort will take 12 months and the driver team will have access to development hardware with sufficient lead time then at-launch enablement is possible. However, some products have a 9-15 month cycle which increases considerably the difficult faced  to build and release drivers with a small team.

Other factors need to be considered if the tempo of release is high – such as feature minimization or sharing code with other drivers that are being developed.

Conclusions

Determining the correct approach for developing drivers is a complex problem. Realistically there is no right solution that is perfect. The possible solution space has been summarized above, but for an individual IHV, the sizeable, but not all-inclusive collection of open issues will need to be prioritized for impact considered in the context of the preferred solution.

Feel free to make comments below or contact me via details on the right.