> In any event, given that Alegeus was aware that DigiCert is required to revoke certificates within 24 hours, one wonders why Alegeus went ahead and signed agreements with their customers that required a lengthy notice period before changing certificates.
It seems obvious enough how this would come about:
1. The healthcare company knows better than to do a Crowdstrike, where "just a configuration change" is rushed out with inadequate testing, for "security". As even a configuration change could trigger a latent bug, they agree configuration changes should be rolled out gradually, and not applied during clients' peak operating hours.
2. The healthcare company could have used "Let's Encrypt" to get certificates for free, if they wanted short-term certificates. But instead they paid $$$$ to DigiCert for certificates with a 12 month duration.
3. Even if Alegeus had read the small print saying mis-issued certificates can be revoked with only 24 hours of notice - they quite reasonably assumed DigiCert would have controls in place to avoid mis-issued certificates. Hasn't DigiCert told the browser vendors precisely that?
> they quite reasonably assumed DigiCert would have controls in place to avoid mis-issued certificates
This makes no sense. The revoke certs when there's been an incident. Telling customers that they might need to revoke certs in the event of an incident should mean that they are going to be free from incidents??
If Stripe tells me webhooks might be delivered more than once, that does NOT mean I would assume they have controls in place to prevent webhooks from being delivered more than once. That's just wild to assume.
My very naive reading is Alegeus may have assumed this certificate revocation would only be from their own screw up - such as publishing their private key somewhere - and they would of course eat some costs on the SLA’s. Instead, we have Alegeus’s vendor saying there was technically the possibility, unconfirmed, that the key could have been compromised, and therefore are executing the nuclear option.
Whether Digicert is liable for damages is another matter (they are and probably should be). What's pertinent is that Alegeus is now blocking that nuclear option.
To make it clear what I mean, imagine this hypothetical conversation:
Alegeus: "We need a certificate with a longer life than the 3 months we can get from Lets Encrypt"
DigiCert: "That is our core business. We will sell you a 12 month certificate. It may be revoked early, if it was mis-issued."
Alegeus: "I particularly need a 12 month certificate, as I mentioned. Is revocation likely? Do you often mis-issue certificates? To your knowledge, are you currently mis-issuing certificates?"
CA/Browser Forum: "Yeah DigiCert, is it likely? Are you currently mis-issuing certificates?"
DigiCert: "We have a number of organisational and technical controls in place to prevent certificates being mis-issued. We do not have a history of mis-issuing certificates, and to our knowledge we are not currently mis-issuing certificates. Our controls have passed third-party audits."
Alegeus: "OK, so to the best of your knowledge as certificate industry experts, this 12 month cert you're selling me will be valid for 12 months?"
To make it clear what I mean, imagine this hypothetical conversation:
Supermarket: "We need a candy bar they we can sell to our customers"
Hershey: "That is our core business. We will sell you candy. It may be recalled, if we discover problems."
Supermarket: "Is a recall likely? Do you often recall candy? To your knowledge, are you currently selling bad candy?"
FDA: "Yeah Hershey, is it likely? Are you currently making toxic candy?"
Hershey: "We have a number of organisational and technical controls in place to prevent contaminated candy. We do not have a history of contamination, and to our knowledge we are not currently contaminating candy. Our controls have passed third-party audits."
Supermarket: "OK, so to the best of your knowledge as candy experts, this candy you're selling me will be good to sell?"
Hershey: "To the best of our knowledge, yes."
If someone warns you about a possible eventuality, you simply don't pretend that the eventuality is impossible. That's why they warned you. Obviously nobody wants that. It's just plain ignorant to have a shocked Pikachu face when the thing they warned you might happen in extenuating circumstances happens.
> 3. Even if Alegeus had read the small print saying mis-issued certificates can be revoked with only 24 hours of notice - they quite reasonably assumed DigiCert would have controls in place to avoid mis-issued certificates. Hasn't DigiCert told the browser vendors precisely that?
I mean, sure, in theory there should be controls in place, but no company or process is perfect. That's why there's a process in place to respond to problems with issuance in the first place.
DigiCert is responding in the way the browser vendors mandate; 24-hour revocation, per the article, is a baseline requirement to be a CA.
1. It does look like there's very strong reason to believe that these particular certs were not misissued, in the sense that the people who have the private keys really are the people who control the domains. Stronger than the sort of verification that the CA/BF requires. It's a bug if the system can't deal with that.
2. It's also true that there are rules, they exist for a reason, these Alegeus guys aren't special, and dragging this stuff into the courts is antisocial in the extreme. So it wouldn't be unreasonable for browsers to, say, explicitly blacklist their domains for as long as there conceivably might be noncompliant certs out there.
> It's also true that there are rules, they exist for a reason, [...] dragging this stuff into the courts is antisocial in the extreme [...] it wouldn't be unreasonable for browsers to, say, explicitly blacklist their domains
You know who else thinks there are rules, that the rules exist for a reason, and that draconian punishments should be applied to people who try to derail attempts to enforce the rules?
Judges. Like the judge that granted this temporary restraining order.
The judge's TRO doesn't apply to browser vendors, it applies to Digi cert. The plaintiff didn't file for an injunction against browser vendors, and they probably couldn't even if they wanted to because they have no business relationship with them. What responsibility do the browsers have to this company?
The judge has issued an order that the websites not be disrupted, just for a few days while the contracts and paperwork get straightened out.
If a browser vendor disrupted the websites, because they knew of this court order and decided to take deliberate action to defy the authority of the court, that's contempt of court.
The browser vendor then gets pulled up in front of the judge who granted the restraining order. The browser vendor says "ah, but the court order only says the certificate can't be revoked, we didn't revoke it we blacklisted it" and the judge says "that's a great argument so I'm not going to send you to jail, I'm going to send you to prison instead"
> IT IS HEREBY ORDERED that DigiCert is prohibited from revoking the security
certificates for the Alegeus Websites for a period of seven (7) days, or until the Court is able to
schedule a hearing on the Motion, whichever is earlier.
The order specifically mentioned Digicert and nobody else. The order doesn't magically apply to everyone else in the world.
The browsers don't have any contract with Alegeus, nor any relationship at all. They owe Alegeus nothing. They're not covered in the TRO. I doubt the TRO could legally be extended to cover them.
Browsers do have a relationship with their users, who may very well be relying on them to enforce the CA/BF validity rules, or if they can't do that to provide equivalent security guarantees. And in particular many of the users rely on their browsers to protect them by blocking suspicious domains in various ways. So blocking the domains wouldn't be arbitrary, capricious, discriminatory, or anything like that.
Judges don't have infinite sua sponte powers to order just anybody to do just anything. Although I admit they sometimes act as though they did, especially when they feel like their authoriteh has been disrespecked.
Anyway the browsers aren't gonna do it. There's zero chance. So legalities don't really arise. I'm just saying it wouldn't be unreasonable.
The issue isn't that someone is taking over this particular healthcare industry domain, just that in that time period there could be bad certs issued for other subdomains and there is no way to verify it. The whole setup where the verification depends on a leading underscore because subdomains aren't allowed to start with an underscore seems rather flimsy to me, but it's how the system is set up.
I see way too many comments here that are basically saying "Why aren't these thousands of contracted orgs able to drop everything at the drop of a hat to update their DNS." Here is my understanding of the issue:
1. Alegeus creates white-labeled sites for their customers, but those sites are hosted on customer domains (usually for ease of sharing login cookies across subdomains on the customer), e.g. alegeus.customerdomain.com.
2. So the issue isn't necessarily that Alegeus can't update their systems, it's that their many customers can't update their DNS records fast enough to point to the new certs.
IMO, if you built a system (and I'm talking about DigiCert here) that expects thousands of customers can update everything at the drop of a hat to update their code, you built a bad system. I don't think it's too different from CrowdStrike, except it's as if CrowdStrike gave 24 hours notice instead, saying "Hey, all of our customers, just a heads up we're going to break all your machines in 24 hours, you really should have planned better if you can't handle that."
Alegeus is the one suing here because they've got tons of customers, but make no mistake this would cause outtages for tons of other companies that use DigiCert.
> IMO, if you built a system (and I'm talking about DigiCert here) that expects thousands of customers can update everything at the drop of a hat to update their code, you built a bad system.
Per the article, this is required to be a CA. It's not a DigiCert thing, it's a baseline requirement. If you can't revoke your certificates quickly, you risk being delisted by Google/Mozilla/Microsoft et al. Entrust was just recently dropped in part because of this very thing.
2 - DigiCert appear not to have put language into their MSAs and so forth that this was a thing that could happen and customers had better be prepared to do it. Why shouldn't Alegeus sue DigiCert to force DigiCert to conform to the terms of the agreement that DigiCert agreed to with Alegeus?
> it's that their many customers can't update their DNS records fast enough to point to the new certs.
This is almost certainly wrong. With the exception of DNSSEC, there's no interaction between DNS and certificates. There's no DNS record that points at a certificate. The HTTP server that terminates the connection is the server that has the certificate.
TXT records can validate a domain, but you can also validate a domain by hosting a file on that domain to show that you control it.
I find it more likely that Alegeus is giving customers appliances or VMs that they run themselves, and those are each configured with a baked-in certificate. Which is bad anyway because it means the customer needs to be involved in the certificate update process.
If you build a system which relies on WebPKI certificates and you don’t have the ability to quickly replace those certificates, your system is insecure and will break. This is fairly central to how the entire concept of a publicly trusted CA works.
Unless you use DANE (which nobody does, and the browsers don't check it even if they did), you do not have to change any DNS records, or touch the DNS at all, to update a certificate. There is no DNS involvement. Period.
Perhaps I am misunderstanding what needs to happen, but since Alegeus itself doesn't control the domains on which its sites are hosted, all of its clients would basically need to rerun this procedure, https://docs.digicert.com/en/certcentral/manage-certificates...
OK, true, that's one of the (inadequate) methods that the CA/BF permits for verification at issuance, and I had forgotten about that.
You can also verify over HTTP or email (egad...). You would think that if Alegeus controlled the HTTP servers and not the DNS servers, Alegeus would opt for HTTP verification, but I concede that they may have built everything around DNS verification.
> that's one of the (inadequate) methods that the CA/BF permits for verification at issuance
Why inadequate (in the absolute)? This can be automated and let's encrypt allows verification through DNS, moreover this allows verification for wildcard certificates.
Now in this particular case maybe they should have gone through HTTP, and even automated with ACME. But there is nothing inadequate in the absolute in DNS verification. Besides allowing wildcard it also allows verification when you don't control the web server(s), when you don't even have a webserver at all, when the standard ports are occupied for something else, etc.
The point of X.509 certificates is that you can't rely on information you get either from the DNS or from the HTTP server. If you could, you wouldn't need the whole mess in the first place.
Sure, the verification helps, because you have to successfully fool both the client and the CA. But if you can fool one, there's a strong chance that you can fool the other. In the end, the CA is still relying on exactly the same information that the client isn't supposed to have to rely on.
The original idea behind X.509 was that verification would be "out of band", but that turned out to be expensive and non-scalable, so the X.509 world, including the CA/BF, resorted to this very weak kind of verification. They try to backstop it with stuff like certificate transparency, but it's just adding epicycles that aren't particularly reassuring.
If everybody used DNSSEC, then DNS-based verification would be OK. But at that point you ought to just distribute key hashes through the DNS and dispense with X.509 entirely. That's actually what should have happened, and probably what would have happened if X.509 hadn't still been such a cash cow at the times when the various standards solidified beyond all chance of improvement. Because of that "cash cow" status, there was a lot of obvious sabotage aimed at entrenching X.509 and fighting any attempt to improve the situation. And now we're stuck with it.
The Chrome team was pretty averse to implementing DANE as an alternative to the web PKI. I don't think I understood why, but their reluctance seemed to be what stopped its momentum (maybe outside of SMTP?).
'agl wrote a blog post about it. There were two big problems, one in principle and one practical.
The practical: you can't reliably run DNSSEC everywhere Chrome runs. Networks get really fucky with any even slightly unusual DNS messages.
The principle: because you can't realistically ever declare a "flag day" and deprecate the X.509 WebPKI, you have to support both systems, so DANE doesn't collapse your trust anchors down to a smaller set; it actually adds to the number of things you have to trust.
I'm more thinking about the pre-Chrome era. The DANE drafts weren't even written, let alone standardized, until it was already hard to move and really hard to get people to move. That slowness had a whole lot to do with obstructionism, FUD, and influence campaigning, from commercial interests, specifically Verisign, and perhaps from some anti-cryptography "national security" interests as well.
Admittedly DNSSEC is an essential prerequisite and wasn't really baked, but it was being delayed by essentially the same FUD. And DNSSEC got quite usable, from a pure technology readiness point of view, pretty fast once people started putting a little urgency behind it later on. It's still relatively hard to use, but that would evaporate in six months if there were some impetus for it to get more users.
Not that the browsers helped, mind you (Mozilla wasn't really any better). A switch to DNS as root of trust still could have been done when DANE was finished if there'd been any real will. Much less will than it takes to establish this or that bad-idea standard that's of no value to anybody but advertisers. Or for that matter to set up weird, mutant, off-the-wall, who-asked-for-that, solving-the-wrong-problem-in-the-wrong-way hackery like DNS over HTTP.
Still, by that time, the browsers could point not only at an entrenched system of practice, but also at a ton of broken, clearly-non-standards-compliant middleboxes on the network screwing up DNS, especially in "The Enterprise(TM)". Personally, I had, and still have, zero sympathy for the people who put those boxes there or put themselves behind them. I would have given them exactly zero consideration. But a lot of people somehow seem to think that one system's bad behavior obliges everybody else to bend over backwards to accommodate that system forever after.
There is actually one legitimate knock against DNSSEC: it makes replies bigger, which makes DNS a worse DoS amplifier. I think it's worth it, and if it weren't my reponse would probably have just been to start moving DNS over TCP. But at least there's a somewhat respectable argument there. Strangely, it was never the main argument you heard.
Ironically, DOS amplification is the one argument against DNSSEC I don't buy; you can already use DNS quite effectively as an amplifier, along with other protocols.
The fundamental problem with this whole line of argument though is that, even if things worked well (they did not; ask Slack) you're still just trading the WebPKI --- with all its warts --- for a system that is even less transparent, and that is de jure operated by world governments. There will, for instance, never be a "DANE Transparency" log; not only because DANE will never be deployed for real, but also because the market forces that coerced CAs into adopting CT don't exist in the DNS.
If there is a contract between DigiCert & Alegeus Technologies, stating that DigiCert could potentially revoke contracts within 24h and Alegeus client contracts state that clients have more than 24h to deal with expired certs, then I feel like it's Alegeus problem, who over-promised to their clients and should have carefully read their contracts.
More to the point, Alegeus should have a system in place that can re-issue certificates within 24 hours since that is the timetable that their registrar works under.
The lawsuit smells like them trying to buy time to fix their IT problems. Even if it loses or gets tossed out all they needed was that preliminary injunction. It's really sucky for DigiCert who have to choose which party to anger, the courts or the browser manufacturers, just because this healthcare company doesn't have its act together.
> The Service Specific Terms are incorporated by reference into this Agreement
> “Service Specific Terms” mean additional terms specific to certain Services as set forth at
www.digicert.com/service-specific-terms, which are incorporated herein to the extent applicable to any specific Services procured by Customer hereunder (which includes the Certificate Terms of Use with respect to Customer’s use of any Certificates).
> The Certificate Terms of Use available at www.digicert.com/certificate-terms (the “Certificate Terms of Use”) apply to Certificates other than Qualified Certificates and PKIoverheid Certificates requested or issued by Customer. The applicable Certificate Terms of Use are incorporated herein by reference.
> DigiCert may revoke a Certificate without notice for the reasons stated in the CPS, including if DigiCert reasonably believes that:
> Industry Standards or DigiCert’s CPS require Certificate revocation, or revocation is necessary to protect the rights, confidential information, operations, or reputation of DigiCert or a third party.
> The CA/Browser Forum Baseline Requirements (BRs) (which all CAs are required to adhere to, by virtue of their being included in various browser and OS trust stores), say that revocation is required within 24 hours when “[t]he CA obtains evidence that the validation of domain authorization or control for any Fully‐Qualified Domain Name or IP address in the Certificate should not be relied upon” (section 4.9.1.1, point 5).
Does that revocation requirement actually apply to these particular certificates?
If I've understood the problem it is that DigiCert was issuing challenge keys for DNS based validation that did not start with the required underscore.
The underscore is required to prevent an attack that only works against sites that allow users to create subdomains.
The particular sites of the plaintiff's clients are not such sites and so has the CA actually obtained "evidence that the validation [...] should not be relied upon"?
How does the CA validate that the plaintiff's client's sites don't match that criteria? Should they take the plaintiff's lawyer's word for it? You can't really go off script with stuff like this, there's too many gotchas that you need to think through.
I wonder if we could come up with a process where a customer could push another DNS record to redo the verification. They'd need to act fast to beat the 24 hour revocation window, but maybe that's better than a full re-issuance? Probably better to invest in re-issuance agility though.
… and the Bugzilla entry where DigiCert reported this to Mozilla:
Digicert pauses mass certificate revocation due to legal action from customer (bugzilla.mozilla.org) — https://news.ycombinator.com/item?id=41114794 — 9 points by frakkingcylons 23 hours ago
The bug sounds interesting. The underscore is needed to ensure the CNAME isn't a valid domain name. For services that offer subdomains for customers, an attacker could get a certificate for the parent domain. Like if example.com offered subdomains based on a name the user chooses, the attacker would start a certificate verification process and get a random value from Digicert. They'd then create an account with that random value, causing the example.com service to create a CNAME for <random>.example.com. Digicert then sees the random value and issues the certificate for example.com to the attacker. The underscore is there as _RANDOM.example.com isn't a valid domain name, so you'd expect services to reject these sorts of domains.
Edit:
> Therefore, this is truly a security-critical incident, as there is a real risk (not a negligible 2^-150 risk as implied by DigiCert) that this flaw could have been exploited to get unauthorized certificates. Revocation of the improperly validated certificates is security-critical.
Yeah... have they proven that this wasn't exploited? With 83,000 certificates getting revoked this isn't a small scale mistake.
> Yeah... have they proven that this wasn't exploited?
Presumably it would be fairly easy to test, for the vast majority of cases.
HTTP file validation is allowed for DV certificates, so you can test a domain by connecting to it. Testing from a variety of locations around the world avoids the risk of MITM. If you issued a certificate for example.com and it's now being used by example.com, doesn't that prove the same thing?
But whether it's been exploited is, in a sense, irrelevant. The baseline requirements don't say "when you detect a mis-issued cert carry out more checks and decide for yourself whether to revoke" - they say the CA must revoke.
Oh sure, they need to revoke yesterday, but if a certificate was mis-issued there's downstream security impact that needs to be handled immediately.
> HTTP file validation is allowed for DV certificates, so you can test a domain by connecting to it.
This was validation over DNS using a CNAME record. You don't need to keep the verification records around after the certificate is issued, so post-hoc analysis won't tell you much. But worse, how does the CA know if a RANDOM.example.com CNAME record was created by a customer vs the domain owner? The underscore is supposed to solve that, and they didn't use it.
One thing that i think is clear from the responses here is that it’s going to be a long time before software engineering is like other engineering disciplines.
> DigiCert knew or should have known about its failure to properly complete the Security Certificates for Alegeus weeks ago.
'You should have known there were bugs in your code!'
> DigiCert is a voluntary member of the Certification Authority Browser Forum
(CABF), which has bylaws stating that certificates with an issue in their domain validation must be revoked within 24 hours
"voluntary" is technically correct, I suppose. But let's assume they'd like to be a CA?
But then we go
> despite that DigiCert has arbitrarily chosen this less than 24-hour window
It wasn't arbitrary, it was a requirement, as the complaint itself pointed out…
> Alegeus has hundreds of such Clients. Alegeus is generally required by contract to give its clients much longer than 24 hours’ notice before executing such a change regarding certification.
Yet another example of that nobody ever thinks about cert rotation, ever.
> Consequently, due to the errors and delays of DigiCert alone, Alegeus cannot practically make all the required changes
Yeah, no, your error was not having a process to account for this in place.
The TRO also states,
> The threatened injury to Alegeus outweighs any injury DigiCert might suffer under the injunction.
Well, if DigiCert is removed as a CA because they've violated the CA/B BRs (at least, AIUI, they have) — that could be a pretty big injury. They're probably too big to fail, and "we tried but we had a TRO" is a pretty good excuse.
The TRO is scoped to only their certs, and only for 7 days (barring further orders), so it's at least pretty minimal? So I suppose DigiCert could revoke the rest.
What is the legal process for "cannot comply with this TRO because doing so would likely cause us to cease to exist"? It certainly is the case that "any injury DigiCert might suffer under the injunction" has the possibility of including no longer being able to be a certificate provider.
The legal process is contempt of court. If CAFB revokes DigiCert than they are interfering with the TRO on DigiCert, which puts them in contempt of court.
The fact that the TRO was issued seems insane to me. So now DigiCert can't revoke the certificates because these jokers can't handle rotating certificates and they're making it everybody else's problem.
Think about what usually happens when a certificate isn't revoked within the 24 hour window. It's not an immediate death sentence, the CABF discusses what happened and decides if they still trust the CA to carry out their responsibilities. The CABF doesn't expect the CA to violate a court order, but there are sure to be questions about how the underscore bug happened, how their agreements should be re-written to reduce the risk of customers filing TROs, and how process has changed. Fixing bugs and re-writing agreements are solvable problems.
I don't think the legal challenges shown here are unique to this CA. A good question for the CABF is how a CA can demonstrate that they've reduced this sort of legal risk. It's a hard problem because there's CAs in so many different legal jurisdictions.
To be clear, I think a TRO is also a pretty good excuse for technical of the rules, and the most likely thing is that people bend to accommodate here.
> how their agreements should be re-written to reduce the risk of customers filing TRO
I actually am not sure a clear agreement matters here. It could be they already have that (I didn't see evidence in the complaint that convinced me one way or the other); if the case proceeds long enough, this I expect will be evident in the ruling: either the agreement was clear, and they should have expected a possible revocation since it is inherently part of the system they're interoperating with (… this would be my default stance; ignorance of certs doesn't get you infinite leeway here) or the agreement wasn't clear at it's a contract breach as alleged.
However, I also agree that, assuming they can rotate in 3 days like they said in the complaint, — but that is an assumption. I mean, IDK about your workplace, but I'm sure you've seen a deadline or ~~two~~ 100% slip? — then the case becomes moot and we'll never know, as there will be no ruling. That could very well happen, we'll have to see.
> I don't think the legal challenges shown here are unique to this CA. A good question for the CABF is how a CA can demonstrate that they've reduced this sort of legal risk. It's a hard problem because there's CAs in so many different legal jurisdictions.
I agree, though I think it is difficult to show that. Like I said, I'm not sure an ironclad agreement saves you from the TRO, here. The judge would still issue a TRO while deciding the agreement is, actually, ironclad, since the judge cannot start from the assumption that it is, and will err on the side of caution.
I think you are right, even if the agreement was perfectly clear, a TRO can be issued. Either the CA system lives with this risk or we only trust CAs that operate in countries without this risk (do any exist?)
Another approach is for the CA to revoke the cert first, then notify the customer. This avoids the TRO as the customer wouldn't know about the planned revocation until it was too late to act.
Does the CAs desire to help customers swiftly move to a new certificate outweigh the risk that a customer would sue to get more time (causing the CA to fail to revoke quickly enough)?
It seems obvious enough how this would come about:
1. The healthcare company knows better than to do a Crowdstrike, where "just a configuration change" is rushed out with inadequate testing, for "security". As even a configuration change could trigger a latent bug, they agree configuration changes should be rolled out gradually, and not applied during clients' peak operating hours.
2. The healthcare company could have used "Let's Encrypt" to get certificates for free, if they wanted short-term certificates. But instead they paid $$$$ to DigiCert for certificates with a 12 month duration.
3. Even if Alegeus had read the small print saying mis-issued certificates can be revoked with only 24 hours of notice - they quite reasonably assumed DigiCert would have controls in place to avoid mis-issued certificates. Hasn't DigiCert told the browser vendors precisely that?