To elaborate, expand and reinforce somewhat what @earthrise just said, and sorry for the delay since the mention, I wanted to get accurate figures for you:
Hey @mistfpga
Thanks for @ing me. Indeed I’m the primary point of contact at ECC for all our external security assessments with commercial vendors. I’ve been planning, coordinating and responsible for our spend in this area since I joined ECC as director of product security in July of 2018.
I’m writing this for a general audience, I know that I’m repeating myself to you and that you know a lot of this stuff already from your experience. I’m just very quiet on this forum and I don’t want anyone to mistake what I’m saying and why I’m saying it. To that end, it’s worth me giving a bit of background on my experience because it’s relevant to the insight that has gone into the way we’ve been buying external security assessments, where they fit into our security program and why I’ve chosen to do things the way I have. If you’re not interested in this part or you already know or you just don’t want to know any more about me, feel free to skip the next paragraph entirely and instead move on to the next one about strategy.
Inserting myself
All I knew until my job prior to this one was security assessment work. Breaking into systems. Pentest, audit, people have different names for it, but basically finding security weaknesses and writing up reports. I started at the bottom and over several years worked up in rank to the higher technical levels through field promotions. Along the way I worked with lots of vendors on lots of contracts, formats, some ad-hoc, some on retainer, some systems (tragically?) were vulnerable in the same ways each time, others required significant amounts of new effort to break into. I broke into middleware applications, messaging, databases, database applications/stored procedures, web applications, network infrastructure, embedded software, door access systems, voip systems, basically anything customers were interested in, so long as it was theirs to attack. I worked with lead developers and system commissioners around the world in their offices and data centers for about ten years as part of several consultancies. Once I worked at a mobile phone vendor on their internal team for a while, and once I worked as a technical risk assessor, translating pentest reports into business risks. In that time the number of systems billed as unbreakable or that a pentest wasn’t necessary, that ended up being flawed and broken was staggering. It wasn’t until March of 2015 that I swapped over to defence full time though. I was the first technical security hire at Fitbit, so I started looking at general security, seeing somewhat more intimately the processes of a company that create security issues and for the first time trying to change those directly rather than just filing them in a report and moving on. I worked with hardware manufacturer vendors on new designs, external security tech vendors, their firmware team, etc. to drive in security into their design at every layer. Great bosses that I’ve had over the years all went on to better things - Google, linkedin, facebook etc. I try to stay in touch somewhat. I am not now nor have I ever been a cryptographer. I have analyzed products for cryptographic security, but only using universal reverse engineering, accepted practices for AES and cipher modes and basic key derivation functions. In short, I’m not a cryptographer, rockstar or otherwise, by any means.
If you got through my life story, hopefully it will be clear where I’m coming from when I say some of this stuff, and the perspective it’s coming from. I don’t believe it to be controversial, but I do feel the need to CYA when posting publicly:
Failure modes of security programs and thinking
There are some strategies that I have seen fail in commercial and open source endeavors with respect to security:
- All you need are pentests.
Having played both sides of this game, I can tell you that I am 100% confident that only having external security assessments in your security program is a quick path to repeated failure. I’ve seen time and again software vendors use only pentests to find the same class of bugs again and again. I know you’ve seen this too because we’ve spoken about it before.
- All you need are bug bounty programs.
This is an extension of 1) and equally wrong. Projects with only bug bounty programs when they get a real pentest get destroyed. Ergo, often necessary but never sufficient.
- All that’s needed is better education.
Education is important but not sufficient. If there’s no reason for someone to implement an appropriately secure system, having the means to do so means nothing.
So really each project, device, offering, system, needs to have a whole security program applied. There have been some attempts over the years to describe what a successful security program looks like, none of which fit any situation perfectly. They don’t fit, much in the same way that the various attempts to quantify security vulnerabilities through some scheme have basically failed over the years. Ref: heartbleed had a CVSS score of 5.0/10.0
I see my job as trying to create and run an entire program for ECC. My superiors at ECC see that job as including describing that program externally - something that was on our Q3 KRs but looks like it will get slipped to Q4 at this point.
Given all the above caveats about the place of external and independent security assessments in ECC’s security program and their usefulness in general, of course they are an important and necessary check on how well you’re doing. One way to think of them is as wellness visits. A bit like going to see the doctor - necessary for good health, but not sufficient. Can provide some useful insight into how well you’re doing.
Finally I’ll talk a bit about the unusually high quality of work that the ECC core engineering team kicks out. This isn’t just me promoting and congratulating their work, it’s of such high quality that it makes a material difference to our purchasing strategy.
Assessments of new core ECC software features typically turn up no materially impactful findings. This is important when we purchase assessments.
ECC’s purchasing strategy
[ Note: If you’re one of our vendors reading this post then know that we appreciate your efforts and nothing here was meant to be duplicitous in any way, but hopefully you see it rather as a solid application of market forces and respect for motivations around pwnage. ]
- We can’t be telling our vendors what to put in their reports.
- We can’t be telling our vendors how long they should be doing security assessments for.
Why? Well we’re forcing them to publish their work, for a start. This is unusual for them, most of their engagements are conducted in secret, used in secret, kept classified by whatever company they are commissioned to work for, and never see the light of day. The same is true for me - my life’s work, basically, is lost to time and kept in darkness. [ side note - computers and security were an escape for me from judgement of the people around me, but now that same mechanism has prevented them from seeing my best work, a cruel twist of irony ].
So working for us is a chance for these vendors to shine publicly, because the reports are published. Which means they have to stand by their work. Which means they’re open to criticism. Which means they’re going to be very sensitive to criticism about how much time we gave them to assess within a given scope.
[ another side note - most people see pentests as “secure” vs “not secure” but for any real system they’re really just a pressure test for how long it could withstand the pressure from these assessors at this time. It’s a limited result and most people don’t understand that. The system under test could have given out completely five minutes after the assessor ran out of time, but we’ll never know. For this reason the assessor’s job is typically to find the biggest problem the fastest possible way. ]
I’ve seen one of our vendors publicly decry the investment strategy of one open source project similar to ours - for dividing their investment between them and another vendor. I think they were probably right in what they were saying, but I didn’t envy them being put in that situation, and I find it difficult to see how that relationship could continue to be productive after that problem arose. We thus far have avoided that kind of fracas. I intend to keep our vendor relationships mutually beneficial for as long as I’m responsible for doing that.
For a while we were doubling up our investment on security assessments on a particular network upgrade. This is in stark contrast to dividing it in half. We did this by contacting one vendor and asking them to use their understanding of our threat model to scope an appropriate amount of time and then quite independently and without telling either party engaging with another vendor to do the same thing. It seems duplicitous, but when you think about it we’re treating each vendor with respect and giving them every chance to succeed.
After we did that the first time, and as part of their general market awareness, they’re then free to select their own scope based on the various competing interests - the desire to defend their work publicly, their desire to promote good work publicly, and their desire to keep us as a client by providing great work at a reasonable price.
We’ve had to cut this down because of financial constraints relatively recently, and now we only have one vendor per network upgrade to save costs. We reserve the right to switch vendors at any time, and we do that from time to time. That risk is still there and is in the mix when they’re scoping work.
Finally, in the case of security assessments that typically turn up no substantive issues, you really don’t want to overstress your assessors. Complacency, learned helplessness and a routine of not finding issues are enemies of great security assessment.
Having the same firm keep hammering away at our stuff and finding nothing that brings them ‘the joy of root’ again and again on retainer seems like a good way to lower the effectiveness of the assessments over time. This is just human nature, not a criticism of our world-class vendors.
tl;dr
All’s to say, we have some particular (maybe even peculiar?) requirements around our security assessment purchasing because of the exceptionally high quality of ECC core team’s work, and the fact that we insist on working publicly. The intersection of those issues is how I’ve concluded that we shouldn’t try to put firms on retainer for this purpose.
Importantly, as I said above, security assessments are really just a check that your security program is working.
In terms of cost, we typically spend anywhere from a thousand dollars to fifteen hundred dollars per person-day for general application security assessment and fifteen hundred to two thousand dollars per person-day for cryptographic security assessment reflecting both the increased specialism involved in that work and the scarcity of services in the marketplace that can perform competent assessments of cryptographic systems.
ECC spent 250k last year on external security assessments, the vast majority of which was for product assessment. We have spent 100k so far this year on external security assessments, all of which have been for product assessment. These numbers aren’t secret, and are published as part of our transparency reports. I believe that we get value for money from these services, despite their rarely finding very significant security issues.
Security proofs, which are a super important key element for our success, and I see as the most specialized cryptographic work we engage externals for, are perhaps somewhat ironically done at significantly lower cost by engaging academic institutions on an ad-hoc basis.