Previously I had a monthly crontab script that updates my LE cert, but now it seems that certbot has been backported to my server's OS (Debian Jessie) which makes my script redundant.
Right this is exactly what the Letsencrypt folks are trying to solve. Thank you for your work Josh! SSL has been a pain in the ass for decades! Generating certs, deploying them, maintaining them. This is why we are not at 100% SSL on the web. The LE folks are trying to fix this. Please send a few dollars their way on their fund raising page to help this goal along.
It's not unobtainable to make using certs MUCH easier and automated, their work should be supported by us all.
Wanted to point out the same. It's sort of marketed towards designers/less technical folk, but even to the seasoned developer it represents potentially less time faffing, and should be viable for at least for MVP or personal projects.
Caddy is mostly just unproven at high scale; theoretically, nginx should be able to handle greater loads, but I wonder what Caddy's limits are and how big the difference would be. Haven't found time to benchmark them head-to-head.
Here's a Dockerized version of NGINX that does just that: https://github.com/occrp/watchful-nginx it also generates the dhparam file if there's isn't one and handles proper log rotations.
disclosure: I work with OCCRP and have submitted pull requests on this.
I'm pretty sure this only targets usage by small individual VPSs or something. For hosting services, especially with multiple nginx frontends, things are different.
If it doesn't block when it can't get a cert, then it becomes very easy for someone to break the security of your site. They just have to block the network path to LetsEncrypt, and then your server will come up without encryption.
But what if it hasn't obtained a cert for the site before (e.g. first run)?
What if the cert doesn't expire in a month -- but only, say, 7 days? 7 minutes? Where do you draw the line? Would you rather have your site go red in browsers or just not have the server start? (Remember, it wasn't started already anyway so it's not like your sites are incurring new downtime.)
Eh, the new run might be a failover, or part of an autoscaling setup, so maybe there could be some downtime? I think the default setting has to be to start anyway, although of course any particular service could be configured not to. Intentional failure seems kind of a blunt instrument?
It's complicated a one-fits-all solution. We ended using the S3 as storage back-end to provision, also to get a single cert for all frontend instances.
We've fully automated Lets Encrypt on our handful of servers and so far it works really well, with only a few glitches. It's specific to our setup but:
-> We do DNS verification through Dehydrated (https://github.com/lukas2511/dehydrated). Because we use in-house DNS running PowerDNS with a MySQL backend, updates to records are a cinch.
-> Dehydrated has "hooks" that allows you to call your own scripts for each step of the verification process. We have an on-prem sysadmin box that does remote administration of the servers (so that none of our servers have direct access to any other server, just in case of a breach), so we just had to update a few commands to work with Dehydrated's hooks.
-> Each remote server has its own directory within the Dehydrated environment, and there's an "update" script that runs from cron that: a) asks that particular server which hostnames it currently has, b) updates the hostname list in that server's Dehydrated directory, and c) invokes Dehydrated.
Because we host some client sites, it felt skeezy to me to be updating files in their web directories on a regular basis even if it's really temporary. DNS updates however are just fine.
All in all Letsencrypt has been a huge, huge win for us. I'm really grateful for both them and for Dehydrated.
1) Why did it take so long for an open source version to come into existence ? (there must have been some serious hurtles Let's Encrypted had to overcome to do this)
2) Given that you can get LE certs for free and use certbot for auto-renewal, why is anyone purchasing certs anymore ?
> 2) Give that you can get LE certs for free and use certbot for auto-renewal, why is anyone purchases certs anymore ?
Let's Encrypt only offers Domain Validation certificates, not Organization Validation or Extended Validation (green bar) certificates [1]. The certificates themselves are only valid for 90 days, which might pose a problem for organizations where their infrastructure makes changing certificates difficult or time consuming. LE does not offer wildcard certificates [1], and there is a rate limit of 20 certificates per week per registered domain [2], so organizations with too many domains won't be able to use LE for all their certs.
This is basically where I am. I have a VPN appliance I have to maintain. It's a manual process to update the certificate. Just about everything else I use runs LetsEncrypt, but this VPN box has a nine dollar cert from Namecheap because nine dollars is totally worth not having to manually deal with the certificate replacement more often than once a year.
> 1) Why did it take so long for an open source version to come into existence ?
Open source tools to run a CA / PKI have existed for a good long time, they just haven't been easy to use.
Automated certificate issuance software has also existed for awhile, but it hasn't been terribly awesome.
The bigger issue is trust - launching a new CA is quite expensive since you need to invest and build all the technology and processes, then get them all audited, and then present the results to the various organizations who control who control which CAs are trusted (see https://cabforum.org/)
The motivation for running a CA has typically been financial - even vendors that provide free certificates (StartSSL, Comodo, etc) have done so as path to moving customers to paid services.
Lets Encrypt/ISRG was kicked off by a group of folks who care deeply about security and had the technical chops and inside knowledge (some (all?) of the founding team) were from Mozilla and had both guidance and experience dealing with the CAB.
In short, there is no good reason not to have done this, and independent of this, another team within Mozilla had started to think about how we could tackle this before we found out the LE work was in flight :)
(Not for ygjb-dupe's interest as I'm sure they know already, but for the sake of the thread)
To slightly expand, getting trusted by the key trust stores (laughably pretending to each represent a "Web browser" at CA/B although nobody would mistake Apple or Microsoft for mere web browser vendors) takes far, far too long to be practical as a first step. Mozilla was fastest and they quote 12-24 months typically, whereas Apple and Oracle are both famously slow to react, and might not sign off on Let's Encrypt for years yet to come.
Well, Let's Encrypt works today, so how did they do that? There's a way to sidestep the trust stores. An existing trusted Certificate Authority can sign a certificate which instead of saying "This is a web server" says basically "This is another Certificate Authority". Let's Encrypt got theirs signed by Identrust. This is called "cross signing". But of course to get an existing CA to sign your CA certificate they need to be sure you're not going to screw up, because it's their reputation on the line as well as yours. So although this is faster than talking to all the CA/B "browser" members for years, it still needed a lot of hard work by ISRG's team.
Indeed, I don't recall atm which one Amazon used for their SSL offerings (which are coupled to AWS iirc), but they used the same approach (while also going through the normal channels).
0. The cost of certificates is pretty insignificant to most corps.
1. They like to have someone to shout at when cert issuance fails. They happily pay money for this ( see point 0 ).
2. LE's cert renewal policies aren't compatible with most corporate change-management processes.
Having someone assigned to perform rolling cert changes every 60 days on every server just isn't feasible or desirable.
When I worked at "Fortune 50 Corp", an annual ( or even biannual / triannual ) planned cert update was a big deal and we had to stage the change across groups of servers in accordance with an algorithm that ensured that if they all inadvertently screwed-up that there enough servers with the old certs to keep production and pre-pro running.
3. Customers, particularly of financial insitutions, like to see the reassuring green EV indicator in their browser.
As everyone is mentioning their favorite tools, I like acmetool[1] in particular. I began to work on a docker/compose/nginx integration[2], but never finished it due shifting priorities. The missing piece was a reload hook for nginx. It would have been easy with simple cgi script (just send nohup), but alas, nginx does not support cgi.
To integrate my acme client with nginx running in a separated Docker container I started nginx together with a shell script that reads from a named pipe and then sends signal to nginx to reread configuration. Then the client writes to the pipe when it gets a new certificate.
That sounds like it would work just fine. I personally took kinda dogmatic approach and tried to avoid having several (long running) processes in a container. As such I think a embedded Lua script in nginx would have been most suitable solution
If you have Docker 1.12 or later you can run your acme client in the same process namespace as the nginx container. Then the client can signal nginx directly about certificate change.
I use https://github.com/jetstack/kube-lego and so far it works . I didn't get the to 90 day expiry yet, but it successfully automatically acquired the first certificate. I heard good things from people who have been using it for a while.
I just remarked the other day that I haven't had an automatic renewal work yet with tiny-acme. In theory, it should be ok, but for the first renewal the intermediate certificate changed, requiring manual intervention. For the second renewal, the Let's Encrypt terms of service changed, again requiring manual intervention.
Alas, I don't really want to use the official client. It seems messy and complex, and docker kind of contributes to that impression. I wish there was a tool that was simple, clean and elegant.
FWIW, both of these issues are acme-tiny-specific quirks. The protocol allows clients to handle both intermediate certificate changes and ToS updates without any user interaction (and without breaking renewal). These things were not implemented in acme-tiny in order to keep the LOC count low from what I can tell.
I tend to recommend lego[1] to anyone needing a high-quality ACME client without certbot's complex auto-configuration features. Might be what you're looking for. It's written in go and can be installed by fetching a single binary.
If you need a solution that works with your existing load-balancer/proxy/L7 router/nodejs server/microservice architecture, I built ten-ply-crest [0]. It's express middleware that is fully automatic to register new domains, when used with Consul and an L7 router (Fabio, Traefik, Envoy, HAProxy) to route to .well-known/.
Ten-ply-crest can run as a standalone microservice or on top of any existing express app. It supports pluggable backends for storage, and doesn't rely on the local filesystem.
It's really short on docs, but "release early and often!". It has been running all my sites since the beginning of the year, Feel free to open issues with any questions and I'll help you get running.
If you need a complete client lib for the LetsEncrypt API in JS, that's included too [1].
Let's Encrypt worked well for me the other day, except that I hit a firewall problem. I firewall all "new connection" outbound traffic, so that if a web app gets compromised, it can't start sending spam or try to spread a compromise to other sites.
This caused Let's Encrypt to fail.
I worked around this by temporarily allowing outbound connections, but I wonder whether this needs to be fixed more generally? It seems reasonable to me to block all outbound connections on a web server that isn't expected to need any. But Let's Encrypt's (perfectly reasonable) need to renew certificates breaks this assumption. Unfortunately, it doesn't seem easy to set up an exception on an IP-based firewall to allow Let's Encrypt outbound connections only. This makes automated renewals hard.
If you're using a local iptables ruleset as your firewall, you can use the owner module [1] to filter by UID/GID, for example allow establishing connections from the user that runs your LE automation.
The common solution to this problem is to deploy a HTTP(S) proxy (like Squid) for outgoing connections. These can operate based on things like the Host header rather than the destination IP. Your web server would be locked down to only allow outgoing connections to this proxy.
No, it's the outbound API connection that is the problem. The server listens on HTTPS from the world, so having that listening inbound wasn't a problem.
Right, so you can run the dns01 challenge on a separate service, up to and including running it in jenkins to automatically renew and push your certs, and kick the nginx server. I should document how my setup works and put it up somewhere...
Have a separate host that handles all of the renewals (e.g. dns01 challenge), then pushes the updated cert to your webserver and reloads the nginx (or whatever process is required for your webserver/proxy).
Is it possible to fix it more generally? You need to make the renewal request somehow, and that means there has to be an outbound request at some point.
It's a good question. I'm not sure. It is, however, a caveat of the Let's Encrypt experience for me - whether or not it is possible to fix.
> You need to make the renewal request somehow, and that means there has to be an outbound request at some point.
Agreed. It would be nice if there were some way to set up an (external) IP-based firewall such that the Let's Encrypt API calls can go through, but nothing else.
One way might be to declare publish destination IPs as static (rather than dynamically through DNS), but of course this would be rather brittle for them from a service perspective.
Another way might be to get a port number assigned for the ACME protocol, rather than overloading HTTPS. ACME services could operate on both an official ACME port number as well as the HTTPS port for backwards compatibility. The protocol could remain HTTPS-based. Then firewalls could explicitly allow ACME through, while blocking outbound HTTPS for anything else that may be an attempt to exploit a general HTTPS server rather than being known to be ACME-specific.
How practical would it be to route the outbound requests through another machine (or VM?) which has more privileges, but is locked down to do nothing but send ACME requests? Or have that VM be entirely in charge of certificate renewal, and give it access to your certificate store somehow?
> Or have that VM be entirely in charge of certificate renewal, and give it access to your certificate store somehow?
This is what I'd prefer, I think. The thing is, part of the appeal of Let's Encrypt is the readily available ACME client that JFDI. Such an easy-to-use client doesn't currently exist for this use case.
I like automating via docker there are various docker images with haproxy or nginx in it that automatically refresh certificates. Just install them as reverse proxy for your frontend and here we are. Very useful and almost no configuration needed.
I love the Docker solution for this. Every day I find more and more problems that Docker can solve for me.
However I dont think this will work for my setup. I run Arch Linux on a Raspberry PI, which serves OwnCloud via Apache. I have always had to go the manual route with lets encrypt because I dont run Debian.
The problem is these docker images are probably built with ubuntu or debian and assume an intel architecture. So I would have to recreate these images for arm - which means I would have to start with a different image.
Not hard to build myself, but I would basically need the original Dockerfile and then recompile from there.
> Don't sweat the permissions for this directory; the certificates themselves will not be publicly accessible.
Maybe still a good idea to sweat the permissions - the way it's set up right now, any process on the system can delete the certificates from this folder, thus breaking your site until such time as you renew them.
Seems like cron really should have a way to run a job at a random moment between two points in time. I searched around some and all the solutions involve terrible hacks like prefacing your command with /bin/sleep $((RANDOM\%3600)).
I think what turns it into "terrible hack" for me is the fact that it won't properly handle the situation where the machine reboots after firing the cron job and before the sleep completes. Having an extra process sitting around for potentially hours doing nothing but waiting also seems ugly. Neither is particularly important though....
Try scheduling a job with "at" in that crontab entry, some systems have it installed by default and jobs persist across reboots. The at jobs are executed through cron themselves.
Put "certbot-auto renew --quiet --no-self-upgrade" in crontab and it does its own thing.