Let’s Encrypt Infrastructure

We occasionally get questions about what Let’s Encrypt’s operations infrastructure is like. Here’s a quick overview.

Let’s Encrypt’s services are operated on dedicated infrastructure with stringent physical access controls. We currently have about 38 rack units of hardware, consisting primarily of Hardware Security Modules (HSMs), compute nodes, storage, switches, and firewalls. There is quite a bit of physical and logical redundancy to protect us from failures.

The hardware is split between two sites. These two sites are separated such that it’s very unlikely that a major event could bring down both sites. At each site, our hardware is located inside a special secure room inside a datacenter. These special rooms require extra authentication, and cannot be entered alone.

We primarily use Linux for operating systems. We make heavy use of configuration management to automate deployments; our goal is that nothing be deployed or configured manually in our environment. We are even working to bring systems not typically manageable in this way under this paradigm. As a result we can re-deploy identical environments in a matter of minutes and there are no surprises.

Our API endpoints and OCSP services are proxied by Akamai. This gives us powerful traffic management capabilities, including DOS mitigation and caching. This greatly increases our confidence that we can keep our services up and running in extreme traffic conditions.

Our infrastructure is constantly under internal review, but we also rely on audits to help ensure safety and correctness. We go through WebTrust audits to ensure that we’re complying with the Baseline Requirements and meeting or exceeding the expectations of the Web PKI community. We also have security audits, including penetration tests, performed by a separate entity. Both audit types provide us with valuable feedback.

Our operations team has worked incredibly hard over the past year to get this infrastructure ready and we’re pleased with the results so far.

Wells Fargo Rewards is a Sleazy Program

Here is how the rewards program works. First my banker bothers me about enrolling until I do. Undoubtedly he got commission for that which ultimately I paid for in higher fees. Then I get charged $19.00 per year for the rewards program enrollment. Over time I built up points by spending money on my credit and debit cards. Then I have to go to a totally different website from the normal WellsFargo.com site, WellsFargoRewards.com, register a new account there, then log in and select cash rewards. This is hundreds of dollars that they’ve been sitting on for years.

They took my money up front and again every year automatically, just for this program. Then they make it non-automatic and non-obvious for me to reap the benefits of what I paid for. The whole process involves wasting my time (talking to my banker about it, signing up and redeeming points) just to get cash I’m entitled to. I don’t want to play this stupid, sleazy game. If they are sitting on all of this money for me why can’t they just lower my fees and interest rates with it? Because they’re hoping I’m too forgetful and/or inept to redeem, but they’ll never forget to charge me the fee!

Due to things like this (there’s more but I won’t bore you with it) and their part in the financial/mortgage crisis over the past few years I’m done doing business with Wells Fargo.

Bye Bye Wii, Hello Canoe!

I sold my Nintendo Wii and bought a canoe with my friend Andrew. We live right next to a bunch of lakes including Lake of the Isles, and they are awesome for canoeing. Turns out I don’t really want to play video games much.

We got a canoe that is pretty wide so it’ll be stable with a passenger and for having on-the-lake picnics. I’m so excited! Off to buy some life jackets…

New canoe
Got a great deal on this Old Town Canoe off of craigslist. Guy bought it for his kids, they don’t use it, but I will!

Break it up, bucko

I started landing pieces of my Mac OS X native theme rewrite. Native theming is not turned on for content in Firefox yet, I’ll flip the switch when the majority of it has landed and things are working well.

Originally I had it in my head that I was going to post a monster patch that just *poof* fixed native theming on Mac OS X and turned it all on. Turns out that this is more complicated than I thought it would be and I was ignoring some lessons I’ve learned in the past about monster patches. Now I’m breaking up my big patch into digestable chunks and it is making things much easier.

Native Form Widgets Getting Closer

I’ve been working hard on new native form widgets for Mac OS X. I am pretty far along and thing are looking great, but I’ve run up against some difficult layout problems that are going to take some more time to solve the right way. It is now clear to me why nobody wrote a kick-ass native form widgets implementation for Mac OS X before. It is really hard! The biggest problem so far is that theme support via Mozilla’s nsITheme API is not very good. Our layout code only implements support for nsITheme in the specific places where Windows and maybe GTK need it.

There are a bunch of other issues with form control vertical alignment and popup button rendering that are specific to Mac OS X and fixing those without hacks is going to be tricky, but I think taking the time to do this the right way is definitely the way to go. If we go with the hacks it is just going to cause more pain in the future, and we might never get around to undoing them. That is pretty much the story of Mozilla on Mac OS X up until now, and it is time to change that.

I’m a little burned out on native theme stuff for now, I think I’m going to take a short break and work on some other cocoa widget bugs.