Lesson 1 rings a mad bell. I distinctively remember a text from a colleague saying "yeah, I'm never launching anything on a Friday".

As a general rule, the best time to launch is first thing on a Tuesday. Why Tuesday morning? We are also well aware of "Monday morning madness" too - pre-planning stress for Monday morning is akin to masochism.

Agreed, we have a strict no deployments on Fridays or after 3pm on any day at work. Allows us to go home with a clear head and enjoy our weekends.

Never really understood person that do otherwise. What's the point ? ruining a perfectly fine night / week end ?

For essentially all companies whose customers are paying consumers:

The technical staff all talk about the dangers of Friday evening deployments.

The company has ignored/overridden these warnings in the past, and sometimes experienced catastrophic failures because of it.

STILL, sometimes the marketing staff will have weekend promotions that are considered more important than these concerns.

In particular, a weekend promotion will be implemented, tested on Wednesday, and ready to go. And only after it's active, is a bug found in it on Friday at 5pm. And the promo is more important than the risk of failure.

I'm not saying this as a "bitter techie"... I'm just explaining that this is how it happens.

I think that the "it's more important" argument is often correct. Not running the promo costs the company $X of opportunity cost, and X is often very well known, from past promotions. Balance "losing $X" against "well, there's only a small chance of something going wrong", and that's why the risk is often taken.

Take the risk enough times, and eventually something goes wrong.

I wonder if this is the rationale for Patch Tuesday.

Pretty much. It's arbitrary, but is fixed, not the first week or last week of the month (often reporting periods), not Monday, and not the weekend. If you then assume that organisations test the patches, that means they can deploy them for thursday morning

No the rationale for patch Tuesday is shoddy development while continually introducing bugs with your fixes.

I work for a Forex (have USD? Want JPY?) service provider that runs a 24-5 service (At any time of day during a weekday an exchange will be open somewhere, on some continent. They all close on weekends)

We exclusively push code on Fridays.

I've worked with a global services industry organisation based in the UK but with MAJOR work happening in the middle east, India and Asia-Pacific regions. To add to that, because so much of the service industry's core activities tend to be based around the weekend and start of the week, we can't simply push at the weekend. Add in regular scheduled downtime and backup periods, having to work with people in the US, and many, many dozens of systems...

This is the difference between a software developer and a senior software developer. I never release anything on a Friday OR the day before a holiday. ALWAYS plan it afterwards, give any technical reason you like (solar flares). Project managers will realise and thank you for it in the long run.

This is definitely something I value about working in finance. Exchanges (the ones we're caring about anyway) are all closed over the weekends.

There is a great lesson in one of pg's essays. If you push you better be there for few hours to monitor what is happening. You do not just push and leave. I hope we all learn from this thread.

In the IT services industry, we'll often talk about running a "war room" for the hours following a significant change, and then follow with a period of a nominated on-call person who has the authority to wake the project team up, tell them to drop everything and fix the issue.

Funnily enough, change control actually runs a lot more smoothly in terms of getting past senior non-technical managers when you include elements like that

It sounds like you don't do continuous deployment. It divides the risk and improves agility. It pushes "change" to a non-management level.

"continuous deployment" is not a panacea to the issue. There's two primary ways of doing it, from what I understand:

continuous deployment to a dev/test environment is the easiest for most organisations to move to. Due to the live environment being mission-critical, they can't afford the risk of any degradation of service. So you push regularly (after passing test suites) to a test environment, get a small subset of users working, and at some point then push out to live from that. But I suspect that isn't what you are referring to, as this is too similar to typical change management.

the alternative definition of continuous deployment is that of constantly pushing to live, initially for a subset of users then rolling it out from there gradually, but always on the live environment. In many large, often 'cloud-y' solutions, that subset might be all users. Except you can't have a public transport ticketing system fail at peak times. A hospital patient record system must stay available for staff, and give sufficient notice before any possible impact to service to allow manual processes to be used. Payroll, accounts, HR systems... all of these have failure modes that have financial penalties at best.

Hence continuous deployment only makes sense when you can afford to risk service, possibly with significant impact.

It helps a lot though. One of many small incremental changes, or one big monster change, which do you think is likely to break a system? And which is going to be easiest to diagnose and fix?

I learned the first lesson the long way. To make myself feel better, my excuse is that it was my first job and I can in on the weekend to fix it.

at my last job we had an unofficial slogan of "don't fuck with it friday"