Reviewing Security Testing Techniques

For vehicle manufacturers and their suppliers, guidelines and standards, and proposed

standards, require the implementation of practical security testing. This is the next stage in the security assessment of a system once the TARA process has progressed. Security testing is an additional overhead in the development process for new vehicle models. Such testing must include the links beyond the boundary of the vehicle because vehicle connectivity has made it part of a wider cyber ecosystem. This means security testing extends to the communications infrastructure and ITS. Fortunately for manufacturers, their existing investments and expertise in functional testing can be leveraged for the challenges of cyber security testing, and, as the J3061 guidelines indicate, testing processes should not need to change a great deal.

Furthermore, if cyber security testing is performed early enough it can allow for feedback into designs prior to production, as it should do to ensure security is baked into systems.

When engineers design a system, they can specify functional security mechanisms, for

example, authentication of a user via a logging-in facility, e.g. entering a Personal Identification Number (PIN) when syncing a smartphone to a vehicle. Any designed-in security mechanisms are to protect the security (CIA) properties of the system. The functional security mechanisms

47_{https://www.isograph.com/software/attacktree/mitigating-against-attacks/}

48_{Oyler, A., and Saiedian, H. (2016) ‘Security in automotive telematics: a survey of threats and risk mitigation strategies}

to counter the existing and emerging attack vectors’, Security and Communication Networks

will be defined in the system specifications. The test plans for the system will check that such defined security mechanisms function as intended50_.

What is often exploited by malicious agents is hidden, and unwanted, functionality51_{, caused by}

engineering issues. In a system that uses software for much of its functionality, it is bugs that cause engineering issues. These bugs can take the form of:

• Logical errors in code (or models used to generate code) resulting in run-time bugs. • Weaknesses in system design, for example, if no consideration has been given to data

encryption or a lack of strong checks to ensure that input can only be received in the expected manner (or format).

• Functional bugs due to a mismatch between what the system specification states and how the system has been implemented, and these not being caught by the functional testing.

• Additional and undocumented features provided by third party components and software libraries. Examples include test functions or features that were developed for another use case (e.g. another customer) that remain present within the system. Not all bugs can be exploited to reveal weaknesses, however, for exploitable bugs, three types of testing can be performed to reveal them10 11_{. Indeed, Section 8.4.7 of J3061 describes them}

as “critical tools in evaluating the Cybersecurity performance of a system”:

• vulnerability testing - performing tests for security weaknesses and exploits using scanning tools and a corpus of known attacks;

• fuzz testing - dynamically sending the system large amounts of random and malformed data to see how it responds, in an effort to reveal a vulnerability;

• penetration testing - using intelligence and tools to attack a system based on how adversaries would attempt to overcome security mechanisms.

As researchers11 _{point out, these three classes of security tests are relatively new to}

automotive software and test engineers, and mobile network and communications engineers. Furthermore, they add to the existing systems functional testing workload. Integrating security tests systematically and rigorously into the systems testing regimes will take some effort, particularly considering the complexity and number of interfaces that can now be found with the ecosystem. The proliferation of advanced features, supported through cloud-based services which can be probed for weaknesses at the client (vehicle or device) or server (service provider) ends, adds to the attack surface. Whilst mobile networks carry out extensive cyber security checks on a regularly basis, the incorporation of mobile communications into the CAV ecosystem extends the attack footprint.

The work required to implement the three types of security tests into CAV testing regimes has begun52_{, and researchers at Chalmers University of Technology have used the three security}

tests as part of a proposed Start - Predict - Mitigate - Test (SPMT) process to systematically

50 _{S. Bayer, T. Enderle, D.-K. Oka, and M. Wolf (2016) ‘Automotive Security Testing-The Digital Crash Test’,}

in Energy Consumption and Autonomous Driving Proceedings of the 3rd CESA Automotive Electronics Congress, Paris, 2014, J. Langheim, Ed., Paris: Springer

51 _{H. H. Thompson (2003) ‘Why security testing is hard’, IEEE Security and Privacy, vol. 1, no. 4, pp. 83–86,}

ISSN: 15407993. doi: 10.1109/MSECP.2003.1219078

52 _{P. Wooderson and D. Ward (2017) ‘Cybersecurity Testing and Validation’ in SAE Technical Paper, SAE}

analyse vehicular cyber security53_{. The SPMT process, which has the high-level steps}

illustrated in Figure 4, is complicated by vehicle complexity and the nature of the possible threats, however, here is a brief summary of the four phases:

• Start - Perform an analysis of the vehicle systems to determine what needs protecting. This is the TARA in the Security Framework.

• Predict - Perform a threat assessment to quantify and rank risks. This would be the application of attack tracks from the Security Framework.

• Mitigate - Apply countermeasures to the ranked risks. Economic considerations influence the application of countermeasures. Mitigation is based upon the outputs of the previous two phases.

• Test - Apply the three security testing regimes (vulnerability, fuzz, penetration) to ensure that designs are resilient to attack, and that applied countermeasures are effective. Use test automation, where possible, for efficiency. Any revealed security issues must be reviewed.

Figure 10. The SPMT process for security testing proposed by Chalmers University

J3061 suggests that vulnerability and penetration testing is performed by parties independent of the systems engineering development. Indeed, BMW proved the benefit of having vehicles tested by independent automotive security specialists54_{. Having an unbiased and independent}

analysis of a system by security experts can reveal unconsidered exploitation paths and system weaknesses.

53 _{Strandberg, K., Olovsson, T., & Jonsson, E. (2018) ‘Securing the Connected Car: A Security-} Enhancement Methodology’, IEEE Vehicular Technology Magazine, 13(1), 56–65, doi:

10.1109/MVT.2017.2758179

54 _{Tencent Keen Security Lab (2018) ‘Experimental Security Assessment of BMW Cars: A Summary Report’}

Vulnerability and penetration testing can be performed by manufacturers, but there may be a bias in the results. Fuzz testing is a beneficial security test for manufacturers to perform themselves. However, a lack of available fuzz testing resources, and information to implement fuzz testing, is likely to restrict adoption. This is one area which would benefit from additional investment into research.

In this section, the major types of security testing have been discussed. Security tests are the practical engineering that occurs to ensure that a level of security, as specified in requirements and designed into a system, is present, and that any mitigation designs have been addressed. The tests form part of the Security Framework, following on from the derivation of the attack surfaces from a reference architecture, the TARA process, and the Attack Tree rankings.

5. Possible Cyber Security Vulnerabilities in