Observable activities are best security metric, panel says

Article

Observable activities are best security metric, panel says

Michael S. Mimoso, Editor, Information Security magazine
SAN FRANCISCO -- The quest for reliable metrics on the effectiveness of information security programs is one that flummoxes organizations seeking a correlation between their activities and outcomes.

Perhaps, suggested a panel of experts at the 2009 RSA Conference, the problem lies in their attempt to measure an abstract such as effectiveness.

"The key is to try to get metrics to be less about quality and more about an activity you can actually observe" said Cigital Inc. CTO Gary McGraw during a panel discussion Wednesday. "We can observe, for example, whether an organization is doing code review -- that's easy. Whether they're doing it effectively is harder."

    Requires Free Membership to View

    Login

    SearchFinancialSecurity.com members gain immediate and unlimited access to in-depth technical advice, strategies, and expert guides for securing data in high-risk financial environments. Join me on SearchFinancialSecurity.com today!

    Michael S. Mimoso, Editorial Director

    By submitting your registration information to SearchFinancialSecurity.com you agree to receive email communications from TechTarget and TechTarget partners. We encourage you to read our Privacy Policy which contains important disclosures about how we collect and use your registration and other information. If you reside outside of the United States, by submitting this registration information you consent to having your personal data transferred to and processed in the United States. Your use of SearchFinancialSecurity.com is governed by our Terms of Use. You may contact us at webmaster@TechTarget.com.

See all our coverage of RSA Conference 2009:

SearchSecurity.com and Information Security magazine editors are in San Francisco to bring you the biggest RSA Conference 2009 news stories, interviews, podcasts, videos and more.
The panel, which included Microsoft Security Development Life Cycle (SDLC) Program Manager Adam Shostack, University of Pennsylvania Associate Professor of Computer and Information Science Matt Blaze, and PlexLogic LLC CTO Elizabeth Nichols, discussed not only what metrics to collect, but also the difficulty in getting organizations to share data in order to build metrics based on actual incidents rather than anecdotes. As Blaze pointed out, organizations are reticent to share data about breaches, for example, for fear of public embarrassment.

"Getting information is difficult," McGraw said. We need a system where information comes to us and we don't have to chase it."

Having data that's been collected over a period of time enables a security program to create benchmarks for itself and observe trends. Nichols compared these observations to what is known as the treatment effect in medical circles.

"You can eventually compare yourself to yourself," she said. "If you spend $200,000 on a SIM, what is the treatment effect of that investment?"

Microsoft's SDLC is one such established program that has specific success criteria, which Shostack said developers and management measures against year after year. The prime metric is to continually reduce the number of security issues shipping in production code, in addition to releasing fewer security updates. Shostack said Microsoft also measures bug counts, the rate at which bugs are found and the software development stage in which they're found.

However, Blaze challenged the notion that success is measured by how often bugs are fixed, suggesting that there is an obvious way to skew that metric in Microsoft's favor. Shostack countered by saying that is an impossibility because of the hacker community's continuous poking of Microsoft products looking for critical and exploitable vulnerabilities.

"There are a set of people who can hold our feet to the fire," Shostack said. "They will shout 'Hey, look what I found and Microsoft hasn't fixed it yet.'"

Why metrics matter

In an opinion piece, Pete Lindstrom explains why metrics can be the unifying language for security and business groups.
The panel also noted the difficulty in transferring successful metrics from one organization to another, and one vertical market to another. To that end, McGraw -- along with Fortify Software Inc. Founder and Chief Scientist Brian Chess and Cigital's Director of Knowledge Management Sammy Migues -- announced the Building Security in Maturity Model on March 4. BSIMM combats this difficulty to a degree, illustrating the effectiveness of measuring observable activities versus effectiveness. BSIMM, McGraw said, was also a great example of the importance and effectiveness of information sharing when done correctly.

BSIMM studied the software security metrics used by nine massive organizations, including Microsoft, Google Inc., Wells Fargo & Co., and the Depository Trust & Clearing Corporation (DTCC), and produced what McGraw called a software security yardstick based on observed activities.

"What works for one organization is unlikely to work for another, even if they're in the same vertical. Investment banks, retirement services firms and the DTCC are all regulated by the same regulators, but culturally, they're different enough so the metrics don't work," McGraw said. "That's why we look at observables."