Listen Now

Data is so fragile. Now that data has moved into the cloud, it’s more accessible, but also more ethereal. We love how easy it is to work with remote data, but we have to be even more careful about creating back up points and fail-safes. Today’s story highlights web-based sewing pattern business Colette Patterns, and what happened when they accidentally destroyed some of their customer data. About Our Guests: Kenn Wilson is the co-founder of Colette Media and oversees the backend technology operations of the company. Apart from a short stint dabbling in spirits and cocktail writing and events, he has worked in web development and systems administration since 2000 as an independent consultant and on the staff of internet companies both large and small. This Episode’s Music Lullatone make sweet, sleepy, sine-wave-riddled songs with whispered lyrics, poppy melodies, and sparse, carefully arranged beats. Lullatone creators Yoshimi Tomida and Shawn James Seymour utilize children’s instruments, splashing water, household sounds, keyboards, glockenspiel, melodica, and electronic sounds to craft their delightful songs for both young and old. This duo truly embody the child’s spirit reminding us all that our bodies may grow old but our souls can stay forever young. Please tell your friends about Hard Refresh, to help us grow. Thank you!

We’re working hard on several episodes coming next year, but we’re taking a month off of publishing, to do season human things. So grab some ‘nog, snuggle a robot, hug your humans, and enjoy this brief message from all of us here at Hard Refresh.

The EMyth Incident Revisited This is the story of how a web development company was really excited to launch an updated version of its own website, and it went wrong in just about the worst way possible. So I was watching this torrent of matrix-like text just roll across my screen, thinking about rocketlift.com because we’re working on rocketlift.com, and my eyes caught emyth.com in there, and my brain went, “Oh, that’s wrong. That’s not the site that we’re working on right now! Something is terribly amiss.” – Matt Pearson Spoiler alert: this is the story of the Rocket Lift’s very first makeover to it’s own website, and the debacle that followed. That’s right. This is a story about us, and how we accidentally deployed code onto our client’s server. About Our Guests Matt Pearson is an avid supporter of open source, open data, open standards, and the indie-web. He also has something of an obsession with information security and the tension between publicness and privacy in our ever more interconnected world. Matt manages Rocket Lift’s server hosting infrastructure — the hard work you typically won’t see — and he consults with our clients on information architecture and critical systems for information security, backups, and off-site redundancy. Before co-founding Rocket Lift, Matt spent seven years in IT at Whitman College. Matt lives in beautiful Seattle, where he and his wife are owned by a couple of cats. For endless hilarity, follow Matt Pearson on Twitter: @matro_wtf. Jed Bickford is Director of Product Development at EMyth. His team is focused on building simple tools that use the EMyth Point of View to enable business owners to create stability, growth, and freedom through their work with EMyth Coaches. He’s an entrepreneur at heart and lives in Ashland, Oregon. Follow Jed on Twitter: @jedbickford. Bryan Teoh is a composer, instrumentalist, and new media artist working out of Brooklyn, NY. His work often explores the relationship between acoustic/synthetic soundscapes, composition/improvisation, and popular/academic music utilizing custom software as well as instrumental work on the viola da gamba, cello, guitar, and piano. He holds a degree in theory/composition from The Lawrence Conservatory of Music where his studies of jazz and classical music were intermittently interrupted to independently venture into electronic music and contemporary studio technique. When not wading into increasingly esoteric areas of musical ephemera, Bryan enjoys cycling, posing as a coffee snob, referring to himself in the 3rd person, and attempting to make the perfect burrito. Learn more about Bryan at sleepfacingwest.com. The songs we featured this episode can be found here and here. Transcript: Intro: [Catherine] More and more, our lives and businesses depend on internet technology. These are stories about the people who pick up the pieces when it all falls apart. Welcome to Hard Refresh. [Matt] So I was watching this torrent of matrix-like text just roll across my screen, and my brain went, “Oh, that’s wrong. That’s not the site that we’re working on right now! Something is terribly amiss.” [Catherine] A web development company was really excited to launch an updated version of its own website, and it went wrong in just about the worst way possible. Here’s your host, Douglas Detrick. [Douglas] Hard Refresh is a production of Rocket Lift, a web development company based in Portland, Oregon. I’m your host, Douglas Detrick, and I’m glad you’re with me. Meet Matt [Doug] So who was this web development team that botched the release of their own website? Well, it was Rocket Lift. Soooo…that’s us. The story we have for you today is about us, for sure, but it’s also about how code that developers write gets from their own computers to working on the web for the whole world to see. The person who does that for our team is Matt Pearson. [Matt] My name is Matt Pearson and I am basically the IT department at Rocket Lift. [Doug] I use the internet everyday, the same as all of you, but Matt is the person I depend on to know how the internet actually works. He supports all of us at Rocket Lift by keeping all our tools working. [Matt] …handling support tickets with other departments in the company, dealing with escalated things with clients, making all of our internal systems like email, our task management system, all the systems we built for our clients, that’s what I do. I make all that stuff go. [Doug] He’s the one who makes the code we write jump from our local computers to going live on the web. And he’s the one who jumps into action when a client’s site goes down. Setting the Scene  [Doug] We’ll get back to Matt soon, but to set the scene for this story, I need to bring in yet another Matthew. This is Matthew Eppelsheimer, Rocket Lift’s Managing Director, and Executive Producer of this podcast. And just to help keep things straight, we’ll always use “Matt” to refer to Matt Pearson, and “Matthew” to refer to Matthew Eppelsheimer. [Matthew] So, it was a festival, it was a party. We were enjoying ourselves. I invited people to bring their own beverage, which was kind of cheeky because it was a video chat hangout. [Douglas] A BYO website launching party over a google hangout? [Matthew] Yes. [Douglas] And tell me about the website you were replacing. [Matthew] The legacy website, our first website, was a very basic, one page, white background, black text. It just had a giant line drawing of a rocket on top. It was something I put together in Photoshop. It wasn’t very good, it wasn’t very professional, but it was good enough for my freelancing work. [Douglas] And who else was on that call? [Matthew] Tricia, our administrator, who really wasn’t involved in the technical work, but was involved in keeping us to our project schedule, just keeping the business running on time. She was a integral part of our company culture. And Tim, the designer, who was really responsible for the look of the site, and the look of our brand. He was the one who designed the logo. [Douglas] What would you say is important about that logo? [Matthew] The rocket itself went through several versions, at least 12 different versions where we were iterating on getting just the right feel for that rocket. The trail coming out of the rocket is actually connected to the “O” in the word Rocket. It’s coming off the circle at a tangent because they launch using the rotation of the Earth as a kind of a slingshot to get them going fast enough to be in orbit quickly. We deliberately did that… It’s kind of dog whistling to anyone who’s a space geek…this is designed by people who know stuff about space and care about not looking like idiots with their logo. [Douglas] So, we’re recreating Cape Canaveral, and we’re launching a rocket. [Matthew] So what was happening behind the scenes was we were playing The Final Countdown, that song from the band Europe. [Douglas] I don’t know it. [Matthew] Well, just so you are aware while doing this… [Douglas] I would love to hear it right now. [Final Countdown plays] [Matthew] You’ve almost definitely heard this before, right? [Douglas] Of course I have!…..Ok, I think that’ll do. [Final Countdown in the room fades out] [Matthew] So that gives you a sense of what was going on. And over the top of that, I was going down my checklist, and I said “Engineering?” and someone representing the developers on the team, probably Tim, said “Engineering is go.” And then probably laughed, because he was like “I can’t believe we’re doing this…” and then i asked “Design?” and Tim would have said “Go!” and so we tweeted design is go, and on down the list. [Douglas] Would you say we were partying like it was 1999? [Matthew] I would say we were, yeah. [Douglas (VO)] This was a big moment for us. If you couldn’t tell, Matthew is and was very proud of this website, and we were and are a very nerdy bunch of people. Don’t Drink and Deploy… [Doug] Instead of 1999, maybe we should say we were partying like it was 1969, the year of Apollo Eleven, and the moon landing. But I digress. All of the team but Matt Pearson, our IT specialist, was celebrating, with adult beverages in hand, because their part of the work was done. They had written the content, done the design, and shipped the code for Rocket Lift’s brand new website. But Matt had still had to deploy all the new code to our server, so he was most definitely not celebrating, at least not with alcohol, not yet. Why wasn’t he celebrating? This story will explain. It was late at night back when Matt was a sophomore in college. A friend’s laptop was on the fritz… [Matt] …and she had school assignments on this thing, and a paper was due and all these terrible things, right? Like, the computer cannot be dead. [Doug] Since he knows more about how computers than your average person, he’s the first call for anyone he knows who’s having computer trouble. If you add to this equation that Matt is an olympic-level nice guy, it means he finds himself fixing computers quite a lot. [Matt] I had some tools to boot the machine up as long as the hardware was fine, the machine would run with my special software, I could copy her stuff across and then blow away her computer’s hard drive and reinstall windows and bring it back to life. [Douglas] He copied her data to his drive, and checked to make sure it worked, and it had. Then as he was reinstalling Windows, it asked him if he wanted to wipe the hard drive clean. He clicked yes, then as the deleting and installing was happening, he thought… [Matt] Something doesn’t look right. Something doesn’t feel right. I think I did something terrible. Cancel. Cancel really quickly. Oh hells. This is not good. Did it work? Oh, it didn’t work. Everything

Richard Tape of the University of British Columbia runs a massive multi-site WordPress installation infrastructure with tens of thousands of individual websites. A change to the WordPress shortcodes API broke hundreds of them, and Richard had to get creative to fix it. The incident would keep him busy with hundreds of urgent support requests for days, and it shook his faith in the WordPress project. About Our Guests: Richard Tape is a WordPress core contributor originally from Manchester, England and now living and loving life in Vancouver, Canada. Checkout his “rant” he published days after his difficult WordPress update. Follow him on Twitter @RichAsInRichard. John James Jacoby is another WordPress core contributor who gave us an inside perspective on the WordPress core update that changed the shortcodes API. He is the founder of BuddyPress, loves puppies and currently resides in Wisconsin. Follow him on Twitter @jjj. The Hard Refresh theme is by The Brow, the latest in a long line of beat projects from genre-bending producer Marcus Williams. His ability to blend the visceral headnod qualities of golden era hip hop with dream pop and indie rock has captured fans across the world. He has been a steady fixture in the Portland scene through the years and continues to remain versatile – these tracks can be uptempo and playful one minute, then gritty and determined the next, but always confidently crafted with a unique penchant for sonic detail. Interstitial music was composed by Douglas Detrick and Wolfgang Amadeus Mozart, and performed by Casey Bozell. The episode was recorded, edited, mixed and mastered by Douglas Detrick, and produced by Douglas Detrick and Catherine Bridge. Thanks to Richard for sharing his story, and to John James Jacoby for lending us his perspective. Transcript: Intro: [Catherine] More and more, our lives and businesses depend on internet technology. These are stories about the people who pick up the pieces when it all falls apart. Welcome to Hard Refresh. [Rich] My team and I checked it all through. Everything looked fine, so we put it into production. It took maybe seven minutes for the first ticket to come through! …my whole site’s broken, nothing works… Theme music drops [Catherine] Richard Tape of the University of British Columbia runs a massive WordPress installation with tens of thousands of individual websites. A change to the WordPress shortcodes API broke hundreds of them, and Richard had to get creative to make things right again. The incident would keep him busy with hundreds of urgent support requests for days, and challenged his faith in the WordPress project. [Douglas] Hard Refresh is a production of Rocket Lift, a web development company based in Portland, Oregon. I’m your host, Douglas Detrick, and I’m glad you’re with me. [music] Meet Richard Tape: What does he do? [Doug] Here’s Rich. [Rich] My name is Richard Tape and I am a WordPress developer for the University of British Columbia. [Doug] …which I’ll refer to as UBC. UBC’s website is filled with the usual information for prospective students and anyone else who wants to know more about the school.  He also runs several separate platforms dedicated to teaching and learning with many websites for faculty and students. [Rich] The biggest one we have unimaginatively titled “Blogs” which is blogs.ubc.ca and it has at the moment 65 thousand sites and about 40 thousand users. [Doug] And there are more. The total number of websites and users here is massive, and Rich is responsible for all of them. Good thing he likes a challenge. [Rich] I got offered the job at UBC and it was too good to turn down. The reason it was too good to turn down was because UBC has been running WordPress multisite for the best part of a decade on campus, and runs one of the largest multisites in the world that we know of. [Doug] All in all Rich manages nearly 100,000 individual websites. The scale of the system is incredible on it’s own, but that’s not all that’s important about it. The websites are vital components of the business of the University. Sometimes students are assigned projects that must be submitted online. [Rich] If a student says “I tried to submit this thing” then we have to actually be able to see what happened. To see whether the student is lying, which is perfectly possible…” [Doug] My digital dog ate my homework… [Rich] “…or, what’s more likely, there was a failure somewhere.” [Doug] The student can say “I submitted this work…” [Rich] “…It was absolutely vital to my degree. Where’s it gone?” And it’s quite stressful! Ultimately speaking, it can go to lawyers. [Doug] Rich didn’t tell us about any lawsuits that he’s been involved in, nor could we find mention of any involving UBC, but it’s obviously something that he’s concerned about. Just because it hasn’t happened yet, doesn’t mean it won’t happen. So, it’s Rich’s responsibility to make sure that this kind of failure doesn’t happen, and if it does, that he has some way to find out what went wrong. [music: theme fades out]  How does Rich manage that many websites? [Doug] There are lots of details that make running a single website a big challenge, so running 100,000 websites sounds, well, impossible. But, It turns out that it is possible with the right tools. Rich uses a special feature of WordPress, called multisite, to make the job more manageable. [Rich] “So, if you are relatively familiar with WordPress in general, you’re probably used to having just one website where you go to the dashboard, and that’s your WordPress dashboard. WordPress multisite allows you to run multiple websites from the same code base. [Doug] Like from a single dashboard. One of the most important parts of maintaining a WordPress website is to keep everything up to date, including WordPress itself, your theme, and your plugins. With WordPress multisite… [Rich] “You can have five websites running from one installation of WP. So, you can have one plugins directory, one set of themes, so that’s only one code base you have to update, when you need to update WordPress or plugins for themes for all five websites.” [Doug] The collection of files, including WordPress, its themes, and plugins, is what Richard calls the code base. It’s what that the server uses to deliver the content of each website to you, the user. To understand how multisite works, you need to know how they work together. So here’s an analogy, with some pretty sounds too! You can think of the multisite system as something like a violin. [violin sound design starts] A violin has a single body—the part that’s made of wood where the sound resonates—and four strings. Each string depends on the underlying infrastructure of the violin’s body to support the strings’ tension, to resonate against and amplify its sound. So, each string is like a different website in a multisite setup, and the body of the violin is like the code base. A problem in the code base will affect all the websites, just like a broken violin body will make all of the violin’s strings useless. And if you think playing a violin with four strings is difficult, and believe me, it is, how does a violin with a hundred thousand strings sound? [violin sound design ends] The risks and rewards of Rich’s method for management [Doug] So there are some risks to using WordPress Multisite to manage this infrastructure. [Rich] If there’s a problem with one of them, there’s a problem with all of them.” [Doug] So, why does he still use it? [Rich] It might not sound like the greatest idea or a huge amount of time saver if you have, say, three or four websites. Yeah, all I have to do is press the Update Now button and it’s done, right? [Doug] Just like Rich says, in WordPress there is an “update now” button that will install updates for plugins, themes and WordPress core. There are more sophisticated ways to do it, but this is how the vast majority of WordPress users do. [Rich] So it might be 10 clicks across four sites. But when you start having hundreds or thousands or tens of thousands of websites, those clicks add up. [Doug] My calculator tells me that 3 clicks per website equals three hundred thousand clicks for Rich. [Doug] So the upside is clear: without a single code base like this, Rich’s job — keeping hundred’s of thousands of sites updated — would be impossible. Rich has a lot of faith in the WordPress platform and the community that supports it. [Rich] WordPress core is remarkably resilient to getting things wrong. [Doug] WordPress is an open source software project. What that means is that thousands of developers  all over the world contribute to improving it. They might find a small issue and report it in the online forums at wordpress.org. And when the next version of WordPress comes out, developers all over the world have contributed to a solution. This community approach to software development means that many skilled people are working together to make sure that WordPress updates cause as few issues as possible. [Rich] Which is one of the reasons we go with WordPress because it is good. It’s really good software. [Doug] So, Rich’s job depends not just on his own skills, but the input of this entire community. And up until this point, that strategy had been working. But soon he’d run into the most serious issue he’d ever dealt with in his professional life. [music] The crisis [Doug] Let’s go back in time to July of 2015. All of Rich’s 100 thousand websites were humming along just fine. And just like he had many times before, Rich received a notification that WordPress had released an update, version 4.2.3. [Rich] “It was a Wednesday afternoon, so I waited until close of play, updated WordPress on the staging environment. My team and I checked it all through. Everything look

Pippin Williamson had a big problem: thousands of business owners were using his software to take payments online, but the latest version was charging the wrong customers for subscriptions. Hear how a coding mistake with data from Paypal and Stripe platforms put businesses at risk, and how they repaired the damage. About our Guests Pippin Williamson is the founder of several widely-used WordPress plugins, including Easy Digital Downloads, and an avid teacher in the WordPress community. Learn more about his work and products at pippinsplugins.com. Chris Koslowski is the Co-Lead Developer of Easy Digital Downloads. Learn more about Chris at chrisk.io. @DouglasDetrick @hardrefreshshow @pippinsplugins thanks for having us on. Loved the episode and can't wait for the next! — Chris Klosowski (@cklosowski) September 17, 2016 Show Notes Find Easy Digital Downloads and the Recurring Payments Extension here. The episode of the Apply Filters Podcast where Pippin and Brad Touesnard first discussed this issue can be found here. The Hard Refresh theme is by The Brow, the latest in a long line of beat projects from genre-bending producer Marcus Williams. His ability to blend the visceral headnod qualities of golden era hip hop with dream pop and indie rock has captured fans across the world. He has been a steady fixture in the Portland scene through the years and continues to remain versatile – these tracks can be uptempo and playful one minute, then gritty and determined the next, but always confidently crafted with a unique penchant for sonic detail. Interstitial music is by The Ocular Concern from their album “Sister Cities,” a danceable and soul-satisfying project co-led by Andrew Oliver and Daniel Duval that ties together West African music, tango, jazz and chamber music. It was produced in Portland, OR, and is released on PJCE Records. Thanks to Pippin and Chris for sharing their story. Thanks to Fastson for typewriter sounds, Dnlburnett for the office sounds, and freesound.org. Show Transcript 0:00 – 0:56 — Intro Intro: [Catherine] More and more, our lives and businesses depend on internet technology. These are stories about the people who pick up the pieces when it all falls apart. Welcome to Hard Refresh. Music begins; overlay Pippin describing the bug and ramifications, stops as the beat drops [Catherine] Pippin Williamson had a big problem. A bug in the latest version of his popular software was charging the wrong customers for new subscriptions. And the worst part about it? It could have been 10 customers, or 10 million. [Douglas] Hard Refresh is a production of Rocket Lift, a web development company based in Portland, Oregon. I’m your host, Douglaslas Detrick, and I’m glad you’re with me. 0:56 – 2:36 — How did this all start? [Pippin] “Well, I’ll go back quite a ways and if it’s too far…” [Douglas] That’s Pippin. He’s the creator of Easy Digital Downloads, a WordPress plugin that allows business owners to sell digital products on their own websites—pretty much anything that can be downloaded. In April 2016 the plugin was installed on over 50 thousand websites, and things were going well with the business. That is until this story starts… [Pippin] “We had two customers come to us and say ‘hey I’ve got this weird issue where a charge has been assigned to the wrong customer. Can you help me figure out what’s wrong?’ In development there’s an idea that if you see a weird problem, it could be a bug, it could be a conflict with some other system they’re running on their site, or who knows what.” [Douglas] Maybe their WordPress software was out of date. Maybe some other plugin was interfering with Easy Digital Downloads. Maybe this was a server problem. Maybe Pippin’s customer just made some honest mistake that caused this, and they didn’t know it. [Pippin] “But if you ever see that same thing a second time, you’ve got an actual bug in your code.” Douglas: And that was the situation he was in: two reports about the same issue. They thought the first support request was a result of random chance. [Pippin] “We didn’t see any logical explanation for why it would happen. And so when logic says there’s no straightforward answer, it’s probably a weird edge case, or a weird conflict.” [Douglas] An “edge case” is a software problem that arises only in a particular situation that the vast majority of users don’t encounter. So, when the second support request came in, Pippin knew this wasn’t just an edge case any more. [Pippin] “Once I had that realization that something was wrong, it took about 30 minutes to identify the bug and figure out exactly why it was happening. Then it was about 72 hours of shitstorm trying to fix it, and freaking out.” [Douglas] You know when you buy an apple, then you notice it’s got a little bad spot, and then you find the whole fruit is rotten beneath the skin? This bug was like that. 2:36 – 4:10 Chris Klosowski was on vacation… [Douglas] One of the things we want to do with this podcast is to show how problems with code online can affect our real lives. Well, here’s a great example. Here’s Chris Koslowski. [Chris] “Pippin and I share the lead responsibility of managing the overall project and direction that Easy Digital Downloads heads in.” [Douglas] When you work in software development, bugs happen. Sometimes bugs happen when you’re on vacation. [Chris] We don’t see our families a lot. So when we take vacations, I’ll take like two weeks and work remotely wherever we’re at as a family, and take a week off when we see my parents. So we were actually in Florida.” [Chris] “I think we had just gotten to her dad’s…” [Douglas] And Chris received a phone call from Pippen. [Chris] “We basically got there and it was like ‘hey, this is going on…” [Douglas] “This is going on” as in “we’ve got a big problem but don’t freak out…” [Chris] “It was bad, but it was one of our best customers that kind of looked at it and saw what was going on, and they were really good at debugging and getting us the information that we needed. We were able to figure out exactly why it happened. That moment was a little bit of panic, but at the same time, that’s the software industry. I’ve been doing it for about six and a half years. It’s not foreign to me that all of a sudden we have to jump on and just start working.” [Douglas] Software developer goes on vacation. Software developer stops vacationing to fix  bug while kids go play on the beach in Florida. It’s a classic tale. 4:10 – 5:45 Back to Pippin: What was at stake? [Douglas] So we know how that moment went down, when they discovered this bug. But what was at stake? Here’s Pippin again. [Pippin] “Well, at lot, frankly.” [Douglas] Remember that the sole function of Easy Digital Downloads and the Recurring Payments extension is to sell products online, and charge customers on a recurring basis. Pippin discovered the product was charging customers for subscriptions that they didn’t buy. [Pippin] “There could be customers that were suddenly paying for someone else’s subscription. There could be customers who had their subscription canceled incorrectly…” [Douglas] The product was charging for subscriptions that customers didn’t buy. The problem undermined the entire purpose of the product, and Pippin had no idea how many people might be affected. [Pippin] “We knew how many customers we had of the plugin. And we knew two or three reports of actual customers that had been affected by it. Beyond that the only thing we could do was speculate.” [Douglas] The way the plugin works is that a store owner runs the plugin on his or her website, and then uses it to sell to their customers. So, both groups of people could be harmed by this bug.” [Pippin] “We know how many customers we have of the product. But what we don’t know is how many customers all of those sites have. We knew that one site had 250 customers. We knew that another site had thirty thousand customers. Who knows? Is there another site that had fifty thousand? Two hundred thousand? A million?” [Douglas] Now, Pippin’s company definitely survived this bug, but at this moment in the story, it seemed like maybe it wouldn’t. They feared they had a huge problem where their customers could be losing… [Pippin] “Dozens, Hundreds, or thousands of customers. We were responsible for that. And we were responsible for companies potentially going out of business.” [Douglas] Destruction on a massive scale. 5:45 – 7:45 — So, What was the problem? [Douglas] When Easy Digital Downloads started it was limited to single transactions, and business owners were asking for a way to sell ongoing subscriptions. So, the Recurring Payments extension was born. But, the first version of the extension was very basic. As Pippin says, it was… [Pippin] “…to be frank, kind of shitty. It was very minimal, it was an MVP” [Douglas] So, they turned Recurring Payments into a much more robust product. But to do that, they needed to completely re-make the plugin. And that’s where things start to get interesting. [Pippin] “So we have a bunch of data in the database that’s a record of the subscriptions that customers have to the site.” [Douglas] Ten bucks a month for this customer, 20 bucks a month for that customer… [Pippin] “The new version of the plugin, in order to offer all the new features and enhancements that we wanted to, required a completely different structure for the database.” [Douglas] But you can’t just do a mass copy and paste—each data point has to be carefully fit into a slot in the new database. [Pippin] “During this upgrade routine we would pull the old data and put it in the new data and there’d be some information that was