Metrics Special Interest Group

Latest updates

This area contains all of the latest information about this group.

• Outputs from the group can be downloaded from the resources area.
• Further information about the group can be found below.

Found 4 articles.

The Challenge

Cyber security suffers from a fundamental lack of clear, objective, universally agreed measurements for the intensity of Internet-mediated threats and the effectiveness of information security systems in reducing these. In the absence of such measurements, it is difficult to determine the relative effectiveness of information security systems and to make objective decisions about how the threat is changing. Most importantly it reduces the effectiveness of information security management, because, as is well known: "you can't manage what you don't measure."

Most vendors of information security products use an estimate of Internet threat level that is based on the concept of potential harm to systems and the intensity of measures that need to be put in place to reduce that potential. For example, both McAfee and Symantec use a four-level system. This takes account of the number and severity of announced software vulnerabilities and known types of circulating malicious code at any one time. The estimate is frequently updated and the level given is based primarily on previous experience. The obvious draw-back to such systems of measurement is that they are essentially subjective. They are also (intentionally) predictive; yet no regular, objective analysis of the accuracy of their predictions is published by the vendors.

In an attempt to introduce objectivity into their measurement of the Internet threat, a number of organizations use 'honey pots' to attract and trap malicious code attacks. The drawback to such systems is that they are specifically designed to be attractive to attackers and are therefore not necessarily truly representative of 'real' systems. Also, because their function is primarily to attract attacks, they will be constantly changed and updated - making it difficult to use them to provide comparative measurements over time.

In addition to tracking vulnerabilities and malicious code, some vendors have also established Internet threat measurement systems that attempt to correlate data from a large number of agents on existing security devices in order to detect developing threats. Examples include Verisign's Cybertrust product, AT&T's intelligence services and Symantec's DeepSight Threat Management System. Such systems more nearly approach the requirement for objective measurement of threat. However, by their nature they must rely on agents installed on existing security devices placed in existing networks. This makes for a very large number of variables: the size of the network; the nature of the organization to which the network belongs; the network's operating system and applications; the location of the security devices in the network; and the type of device being monitored, to name a few. Anyone with experience of statistics will immediately understand that the greater the number of variables the more difficult it is to generate results that are susceptible to good comparative analysis.

It follows that, since there is no clear, objective methodology for measuring and comparing the size and intensity of the Internet-mediated threat, there is no clear and objective methodology for measuring the effectiveness of the countermeasures against it. This lack of ability to quantify the effectiveness of solutions makes it difficult for users to plan cost-effective security and it makes it hard for suppliers to demonstrate the value of security. The objective of this group is to remedy this capability gap by developing prototype metrics and specify a test-bed for validating them.

The Output

The Metrics Special Interest Group (SIG) has therefore been set up in an attempt to meet this challenge. At this early stage we see the following four outputs from this SIG:

1. A specification for a structure to objectively measure the size and intensity of Internet threats and the effectiveness of counter-measures against these.
2. A specification for a 'test bed' to examine the effectiveness of the specification defined in output 1.
3. A description of the resources required to build a functioning test bed as defined in output 2.
4. A description of the interface between a functioning test bed, as defined in output 3, and the wider requirements for information security risk management systems.

The Approach

The Metrics SIG seeks membership from those in industry, academia and government who have an interest and expertise in Internet-mediated threats and / or performance metrics. Membership is especially sought from those who will bring a business-oriented approach to the subject and who are keen to provide functional solutions that will deliver value to real organizations. Participation from vendors is welcome. However, the SIG is seeking vendor-neutral solutions and any vendor representative who seeks to use the SIG as a marketing platform may be barred from participation.

The Threat Metrics SIG will be primarily a virtual group and its work will be conducted through the medium of online discussion. Regular conference calls will be organized, which will also involve the presentation of online material ('Webinars'). As with all the Cyber Security KTN SIGs, Metrics will work to tight deadlines and over a short timescale.

The Chair

The group Chairman is Jeremy Ward of Symantec.

Return to top of page

• Overview
• Introduction
• Activity Group ideas
• 08 Competition Winners
• Overview
• E-Crime prevention
• Outsourcing and Offshore Risk
• SOA Security
• Special Interest Groups
• Overview
• A.I. and Forensics
• Economics of Information Security
• Metrics
• Trusted Computing
• Secure software development
• Privacy
• Human Vulnerabilities