Introduction
Open Source Software (OSS) has been available for a really long time – the original Linux kernel dates all the way back to 1991. Since those early days, the availability of OSS solutions has exploded, covering code libraries, tooling and whole applications. If you have a DevOps strategy in your business, you will most likely be delivering it using OSS tooling.
As we become more aware of the risks of integrating externally sourced software – e.g. the Solarwinds attack in 2020 – we need to be better prepared to manage the risks and responsibilities involved.
Why Use It?
There is strong messaging about time-to-deliver here. OSS has been developed, tested and curated by teams of skilled technicians – often in commercial organisations. Whilst the software continues to be developed and bugs to be fixed by its maintainers, this can be a great source of source code, software and knowledge.
If we needed OpenSSL functionality, for example, would we have the skills to develop a reliable and robust solution in-house? Maybe. Could we do it inside 12 months from a standing start? Nope.
Reasonable considerations are:
- Do you have the skills and knowledge in house to develop a solution yourself to achieve the same functionality?
- If you do create this in house, what does this do to your development timescales?
- If you have to bring in external experts to help develop your own solution, how does this impact your ability to support this going forward?
All of these speak to “time to deliver” for your business application solutions and your ability to support them.
Why do you care?
You should care. You will be very likely to be running Linux in your environment. You may well be using OSS tooling (Putty? Jenkins?) and are highly likely to already be using OSS libraries (glibc, openssl, log4j, etc).
Time to market is a very strong driver, but do you know where all of your OSS comes from? Which versions are in use in your business? What vulnerabilities (CVEs) exist in those versions, and how can you manage them?
Using OSS in the enterprise is hugely beneficial, but do you understand how to manage these risks?
What are the challenges?
Bringing it in
Bringing source code and OSS applications into your business is likely to be non-trivial. Enterprises usually have a restrictive policy with respect to internet access – in part to prevent the loss of business sensitive information, and in part to make sure that staff don’t spend the whole working day watching cat videos!
Before OSS content can be provisioned, the external repository must be mirrored into your network. How do you decide whether or not to trust this repo? Do you trust the whole content or are there very specific items that you will accept from it? For example Docker Hub?
Staging requested OSS to a local mirror, and then scanning for vulnerabilities before making it available in the internal network is one approach. This staged availability also presents an opportunity to manage some other important meta data aspects to capture responsibility and versioning information:
- Who requested it?
- What version was requested
- What was the result of the vulnerability scan
- Established fingerprint (e.g. SHA)
The last of these is important to ensure that versions of ingested OSS can be tied to deployments even where the elements are renamed, whether accidental or deliberate. Without this, building the use profile is unreliable, and this makes fixing vulnerabilities that are identified after deployment becomes much harder. E.g.
- And OSS code library is ingested and used in an application build
- The application is delivered through the route to live and deployed into production, passing all functional and non-functional testing
- A CVE (Common Vulnerability and Exposure) is later identified for the library that is potentially high impact for your business
What is your exposure? Fingerprinting and OSS library version tracking during the build process would allow you to quickly identify the code that would require amending, and fix the problem. But without this, would you even know where to start?
Understanding the Quality
Fundamentally, we need to understand the software that we are introducing from external sources. Complete applications delivered from the OSS community may rely on a trust profile – do you trust the authors? Is it largely being delivered by a “known” entity (e.g. Fedora Core from Red Hat)? Can your business become involved in the project / product development community?
There are other methods that can help establish this trust:
- Use of code quality scanners to establish the application quality of the source code and identify existing vulnerabilities.
- Building the published source code, rather than relying on the pre-built binaries would ensure that there is limited exposure to a mismatch between the source and binary offerings
- Reading and understanding the source code. There really is no substitute for this one!
To Fork, Or Not to Fork?
Community developed and managed code can bring new techniques, technologies and methods to your business, but understanding that this is all lead by the community is important. Because different people may be driving the project in different directions, you may find that the one that your OSS is heading in does not align with your needs.
With a good understanding of the source code, you may be in a position to take up the project direction that you need to support your business by forking – creating a second development community and code stream with a different focus or direction than the main team.
This is not a small commitment, assuming that you wish to retain control of your fork and maintaining the focus and direction will require careful management and some good people skills!
Tracking Use
Understanding which applications are using which versions of OSS within your business is extremely important. Whilst this feels fairly trivial when we’re talking about a desktop application or tool, things become more muddied when considering code libraries. These delivered functions will be baked into your own bespoke application solutions, and the versions that you are using are not necessarily immediately obvious.
Why is this a problem? As mentioned above, if a new CVE is raised against an OSS library version, do you understand the exposure to your business? How can you identify the applications that require patching to close the risk? If the library version is captured in the application bill of materials (BOM), then there is a good chance that you can discover what your exposure is. If the BOM information is stored in a database of application delivery data, this could be understood very promptly with a simple database query.
Gathering and maintaining this data is vital to avoid potentially serious risk. The challenges around handling the Java log4j vulnerability (CVE-2021-44228 – December 2021) where the library was extensively used in Java applications and third party vendor solutions were extensive for some customers.
Support
Answering the requirements of the business for technical support for OSS software can be challenging. If the application is business critical – whether from a business operational point of view or because of regulatory needs – there will need to be support in place.
OSS projects typically have a bug reporting and “best efforts” approach to diagnosing and fixing problems. If your needs are greater than this, then you will need to engage with other options, e.g.
- Your own team. This is easier to achieve if you have a good understanding of the code that you’re using. It may be that you’ve forked the project and have a team of your own people who are now maintaining the fork that you’re using. If not, however, you will need to skill-up your support team to provide this cover.
- Commercially supported versions. Some OSS projects fork into commercial offerings cf MySQL (Oracle) vs MariaDB (OSS), PostgreSQL (OSS) vs EDB (EnterpriseDB).
- Third party support businesses. These can be a good cost effective way of managing your risk, but clarity about how your support needs are prioritised in the event of a larger scale problem impacting multiple customers is vital.
Ultimately, the person on the hook for answering the support challenge and the fallout after the problem has been resolved is you, so careful selection of the best option is hugely important!
Conclusions
OSS will be in your enterprise already. We’ve talked about some of the risks and how we would consider mitigating them.
The value of OSS is the flexibility that it gives you, along with an easier introduction to new coding technologies and above all a solution that you can tweak to meet your specific business needs.
Hopefully, we’ve given you some things to think about to make your engagement a secure, reliable and rewarding one.
If you need support with your DevOps project, contact us info@triton.co.uk