Q- Can you tell us about the structure of your team?
We have three teams based in Poland, Mexico, and CA. Since the teams span two different time zones, user needs can be addressed whenever since we have people available during all hours.
We also use a buddy system. Several engineers at CloudApp have previously worked with each other in past settings, some have even been housemates. It helps us work together effectively when we already know each other, and can hold each other accountable for our work.
Q- What tooling do you use for collaborating with other developers?
For pair programming we use a tool called Screenhero, bought out last year by Slack — it allows you to write on someone else’s screen. And of course, shameless plug…, we use CloudApp for other visual communication. It is much easier to communicate steps to reproduce bugs with screen recordings, especially for UI/UX related bugs.
Q- How do you keep developers aware of customer requirements given that your team is spread across the globe?
Centralizing customer feedback is imperative. We use a tool called Delighted to gather it, before printing it into Slack for the whole team to see. This keeps a consistent thread of customer feedback flowing in so developers and product managers know what new features they should be prioritizing.
Q- Have you set any ground rules for an entire team working at home?
Knowing the potential dangers that can stem from isolating one’s self, every member of our team is encouraged to get out and participate in the community. Whether that means getting together with other programmers or volunteering at a homeless shelter, doing so helps us connect with others on a human level.
Personal learning is also encouraged for the team. The developers are expected to spend 15-20% of their time learning new things that they find interesting… as long as it is somewhat useful to CloudApp.
Q- What’s the best way to nail communication in a remote situation?
I highly recommend the use of emojis, and am the first to acknowledge the seeming silliness of the suggestion. Because it’s difficult to understand tone when speaking with someone over a computer screen, an emoji goes a long way to convey a pleasant, calm tone. This is especially the case when two coworkers do not know each other very well.
Q- What is the team doing in terms of logging?
We use a tool called Fluentd. We typically mount a given appication’s logging directory to a linked Fluentd container — and that container pushes those logs upstream to a cluster of high availability nodes. Changing the final logging repository is just a matter of changing whatever plugin exists on the upstream cluster.
Q- Is there anything you are doing DevOps-wise to reduce downtime and reduce synchronous communication.
For a lot of our new services, we are using a framework called Serverless to deploy code to AWS lambda. Not having to worry about a box failing is really helpful when most of our debugging with other developers is asynchronous.
Q- What are team rituals for staying focused and productive?
Tools like RescueTime and StayFocused are good ones for keeping me in check. I also use SpaceMacs — emacs with evil mode… super awesome, it keeps me super productive. I choose not to use social media at all since it’s too much of a distraction. I’ve created “safe spaces” that are off-limits for programming in order to respect the division between work life and personal life.
I can’t highlight enough the benefits of diet and exercise. I’m also lucky to enjoy the company of my housemate’s dog as I work from home all day. He’s pretty cool.
I’d also suggest some good caffeine supplements with l-theanine, going out for meals, and regularly participating in your local community.
At CloudApp, we are continually striving to find the best ways to provide a great customer feedback led product and engage with our community of millions of users. We would be nothing without our CloudApp champions and love to hear from power users who love our product and benefit from it! If you would like to be featured as a power user, fill out the form below.
Please find the full transcript of the video below:
Thank you everyone for coming. Quick show of hands, how many have, in the past, or currently work remotely? Hopefully I won’t get too many crickets tonight. Okay, cool. If you haven’t noticed already, this talk is about remote engineering in CloudApp, and how we stay productive and mostly sane. Can’t speak for everybody, but I think most of us are mostly sane now.
Here are the things we’re going to talk about. We’re going to talk about team organization, communication, the tools that we use, how we keep our sanity. Also, going to dive into some of our tech stack, at least the parts that I like and the parts that I think are relevant to helping our team work remotely and be productive. Then I might do a quick demo. I threw something together right before this so I’ll probably ask to see how you guys are doing, see if that makes sense. Cool.
First I’ll introduce myself, even more than I was already introduced. I have a background in computer science. After that I was Googling for about two years. I left because the company that recruited me said I could work from home. I decided to move back to Santa Cruz. I lived there and living in Mountain View, no offense to people who live there, it was not very entertaining. There’s not much to do there. It was not worth the move back to Santa Cruz. That company wasn’t so great so I left and joined a company in Santa Cruz and then that was a spectacular failure. After that I joined CloudApp and I’ve been working remotely for them for the past year, about.
Okay, so currently we have three teams. Well, we have one team. We have three locations who are all engineers. We split the team into two squads, one squad for each timezone. In one timezone we have our team in [inaudible 00:01:57]. We have five engineers there. Two remain front-end and then two are mainly back-end. They do growth development. Then we have one who does, I think [inaudible 00:02:08]. Then in Colima, Mexico we have three developers and one PM. It’s really nice having a PM over there to oversee the work that they do. He’s also working [inaudible 00:02:23], but anyway the engineers over there, we have two Mac app and IOS developers and then we also have one [inaudible 00:02:30] guy. Then here in the States we have Daniel. He’s our VP of Engineering. He lives in Los Angeles right now. He just moved back down there. He was living up here for a little bit and then [inaudible 00:02:42] Santa Cruz. Then we have our data guru in whose name is Harts and he’s here in San Francisco.
Yeah, if you find that photo online, there’s a whole string of them and they get more and more interesting. We’ll just say more and more progressive as they go on. One thing that works really well for us is buddy system. I know [inaudible 00:03:12] programming in particular. A lot of what I’m going to talk about here is incidental. A lot of the hiring that we did, we hired based on the buddy system. In order for, in my opinion remote engineering to work really well, you have to have people who have worked together in the past or at least know of each other. In Colima, we have people that worked together. They worked together in the past and they’re part of the same community.
Daniel and I live in the States here. We’ve worked together at companies before. We lived together for a year so we’re really close and working together makes sense. We’re less likely to get mad at each other for [inaudible 00:03:46] because [inaudible 00:03:46]. Same goes for Poland. I’m not sure if they were friends before, but they’ve been working together long enough at this point to where I can say that the buddy system has applied to them. I think for us that’s really important and it’d be hard for me to imagine an organization, at least Cloudapp to be successful without having the buddy system in place. If you’re considering remote work, I would insure that you avoid program isolation and buddy up if you can.
Team sanity is really important. Like I said, isolation is very dangerous. There’s a lot of things we try to encourage our engineers to do. One thing is participate in your local community. It doesn’t matter really in what capacity. It could be programming. It could be just helping out at a homeless shelter, what have you. Just don’t isolate yourself. That’s really dangerous. In terms of programming, isolation can be really dangerous because you could be doing things that are wrong and nobody’s there to tell you. You could develop habits that are actually really harmful. I know a lot of people that work remotely that end up developing really awful [inaudible 00:04:56] programming habits. It’s a mess and it takes a long time to remove those habits after you’ve established them.
We also encourage our teammates to do self-learning and we also try to have like, 15-20% of the time dedicated to things that they actually like. We try to have that dedication a little bit focused towards things that [inaudible 00:05:18] but it’s like, “Okay. I want to [inaudible 00:05:21].” We try to encourage learning that’s [inaudible 00:05:24] helpful [inaudible 00:05:26] because doing the same thing over and over again is not terribly exciting [inaudible 00:05:30].
Next I’m going to talk about communication, collaboration and feedback. When I say feedback, I mean more customer feedback. When you have remote teams, it’s … yeah, Tyler will like this, this is his thing. He’s all about impact scores. Yeah, so when you’re remote, well let’s talk about this first. In terms of the tools we use, it’s a pretty standard affair. I’m not going to talk too much about this but I will mention how many of you recognize the bottom right symbol? It’s called Screenhero, they got bought out by Slack about a year ago. It’s really good for peer programming. I’m not sure you can sign up any more but if you know somebody who has it they can invite you. It’s super awesome because you can actually type in somebody else’s screen so you can compare really well. I don’t think Slack [inaudible 00:06:22] do that [inaudible 00:06:24].
[inaudible 00:06:30] We use excessively as a development team. It helps our QA people to tell us when things are broken. So, when something is broken, say we’re their front end they can just take a screen picture or a screen capture of it and send it to them, or include it in their journal for the day and that’s really helpful for us. Visual communication is a lot more powerful that the driven steps to reproduce and it’s like, I can’t reproduce this how did you do it? It’s really easy to communicate visually with our app so I really recommend it.
Like I mentioned before, it’s really important to centralize your customer feedback. We use a service called Delighted and then we print that to a Slack channel that everybody can see. This gives us direct feedback all day long, all the time, so we can see what people want and prioritize what features we’re going to release. This is important for our fulfillment team because there is no way for them to interact with customers directly or attend customer meetings. So, as you know, it’s important for engineers occasionally to attend customer meetings so they can get a feel for what the problems are. This just really helps out fulfillment team and everybody figure out what needs to be done.
Sometimes the feedback is not very helpful. Like in the bottom of the dreaded iCloud user. For whatever reason people think we’re iCloud and then they complain and say, “How do I put this on the Cloud?” And backup their photos.
More of our customer feedback. Arbitrary graph. One more thing about team communication is to be expressive. Depending on who you’re talking to, if they don’t really know you personally or they’ve [inaudible 00:08:19] you should use as many emojis as possible. I know it’s kinda stupid but if you’re buddies with a person it’s not really necessary [inaudible 00:08:28]. Yeah, I try to use as many emojis as possible.
Moving on to our tech stack. We have a big juicy hamburger as [inaudible 00:08:43] application. It’s written in riddles. We’re moving more towards this to assist our story. I’m not going to into too many details about our stack because TR, DR, VR are married to Amazon so a lot of it is just trying to say we are married to Amazon.
Bits I will talk about are logging, which I really like. This is a system I made. So how many of you are familiar with Fluentd? It’s super awesome. Basically, most of our services, is we containerize them and we develop the logging directory over to the Fluentd container and then we pipe those upstream to our iMobility servers and then those can be piped upstream to whatever plugin you choose. It’s really easy for developers to pick this up into [inaudible 00:09:39] their logging because it’s centralized. The last bit right there, you can pretty much pipe to wherever you want and making that change is really simple. You just change the plugin. That’s all. I find that really helpful. It’s really nice updating this because we have Failover, so if your primary is being updated, you can update that whole thing and then everything will Failover to the secondary.
This is assuming you know how connections [inaudible 00:10:02] on your main setup. If you do, then it won’t Failover but when it does fail over, it will go secondary and then when once it’s back up again everything will switch back. So it’s super easy to update, you’re not going to have any downtime or loss and your developers can pick this up super easy. You don’t really have to do anything extra, like I said, it’s just melting into the log [inaudible 00:10:24] to that container and then it will just work.
Another thing that I really like that we’re starting to adopt is our “Server Less”. I used quotations because obviously there is a server. How many of you are familiar with Server Less or [inaudible 00:10:39] or anything like that. Yeah, it’s been really helpful since a lot of our legacy stuff is ported over to a multi-container environments that requires a lot of [inaudible 00:10:51] and a lot of attention but once we do this kind of server less stack … it’s super easy for us to help developers, producers application and just fire it off and forget it. You don’t have to maintain it, you don’t have to worry about it, you can develop faster and it’s super easy. So, there is one thing about the buddy system that I didn’t mention which is sharing of services. This is more along the line of remote work. What we do is, if somebody develops a service, we make sure that somebody across in another time zone has the knowledge of sharing of service.
So when you make a service you buddy up so that impacts our choice of languages. If you were pick Elixir you have to make sure somebody in the other time zone knows Elixir because if there’s a downtime event while you are asleep cause you can’t be up all the time then you need somebody to be able to at least know how to push the training button or at least know a little bit about the language to get the servers back up and running. Otherwise you’re going to have a huge downtime event which would be really bad. We have [inaudible 00:11:52] sleep and the [inaudible 00:11:54], I guess, so we had an outage for about an hour which was more like two hours. That’s not good. Fortunately it was like three in the morning.
Could do a little bit of a demo. It’s a little bit interesting. I don’t know, I probably shouldn’t be asking you if you want to see a demo but are you curious to see Server Less [inaudible 00:12:17]. Cool.
This is really weird coding links. Alright I got it. We’re there now.
This is a project I set up with the Server Less framework. This is basically something you can deploy to Lambda. So what I’m going to show you is, really quickly, the code that’s required to post up the service to Lambda and basically what you have to do to maintain it. So, this is an existing service that I’m working on right now. It’s not complete, it’s kinda a boilerplate for it. So I’m just going to show you how easy it is to launch a service to [Lambda 00:13:47] via Server less. This is using textures. Are you guys familiar with textures? Yeah. It’s kinda cool, I’m not sold on it so far I still kinda like doing different types and all that kinda stuff but it’s been interesting.
What you need … this is the Server Less manifest so basically you’re specifying what function you want and the function corresponds with an endpoint. So, this [inaudible 00:14:15] this, so it’s basically again for [inaudible 00:14:18] collections. Collections being the object of [inaudible 00:14:21]. And that is comparing to a handler. The handler you define just like normal JS code. So you have handlers here, just lists, so far. Then you look at the handler, it’s pretty basic. You’re making some calls to some service that going into the database through a pothole pulling out the collections returning to the calling.
The tests, their written in Jest. if you’ve ever checked out Jest it’s pretty awesome. It’s pretty easy to use, it’s a lot easier than Mocha or Karma, in my opinion. Or Chai, as well. So, I’m ready to do the install.
We’ll run some tests. It’s going to fail. I did that intentionally because here’s another shameless plug. Typically when we’re working together we could say, “Hey, I’m going to copy all this stuff”, but we also just capture it and then typically our debug site goes, “Hey, I can’t this thing to work. You include the short link … ” Hold on a second. We’ll ask Slackbot for help because I know I don’t have any embarrassing conversations in there.
This is a short link. They can take a look at what’s going wrong. Typically I do this with my terminal instead of copying the code directly because it’s got fancy colors, there’s contrast they can look at and be like, “Hey, that thing is red. I can tell what’s wrong with it.” It says, [inaudible 00:16:23]. Yeah, so it’s not working. So, let’s see here. Fix this. I know that I didn’t activate my environment so I can’t click on the database. Let’s make sure my model is, IT connector is right. Oh, yeah. You can use Jest for the Server Less stuff, which is just a function. There’s nothing to write or [inaudible 00:17:13] in there. It’s a good way to way [inaudible 00:17:15]. You just write a function and you define it where you refer to it in your manifest. That’s super easy and then this thing’s ready. If you want you can just say, Server Less deploy, and we are going to go to … this is going to fail mainly because it has no idea [inaudible 00:17:42] processes. It’s a new convention so I’m going to cheat a little bit. [inaudible 00:17:55]
[inaudible 00:18:00]So, we are going to deploy again. So, assuming [inaudible 00:18:18] with Lambda it will really come down to how much traffic you’re going to see and if you’re throwing in [inaudible 00:18:23] in front of it. If you are [inaudible 00:18:26] in front of it, it can get a little costly if you’re seeing a lot of traffic. It might make more sense to just throw a little [inaudible 00:18:31] in front of a dedicated server. Typically if it’s a service that gets medium traffic, that’s kind of an arbitrary [inaudible 00:18:40] or whatever you want to call it. [inaudible 00:18:43] you get the idea.
You can use Lambda pretty effectively. After I deploy it sends out the endpoint for me automatically. You can see right there, that’s the endpoint, so that’s it. It’s already up. Very simple. Very easy service and it can go right away. [inaudible 00:19:04] and all that. I’m not going to ping that because I didn’t set up RDS [inaudible 00:19:10]. Okay. So that’s it for the demo.
I lost my slides, I think. Oh. Okay, so I am going to close this out by talking about myself because, hey, why not? The things I do to keep my own sanity, obviously … I work from home so I work alone. I have my housemates dog around me all day while she’s at work. He’s kinda cool. But there’s a lot of stuff I have to do to keep my own sanity. Obviously working from home is really awesome, especially [inaudible 00:19:56] but it can get a little bit crazy. Mostly just good exercise, good diet, I’m sure to go out to eat, meet with people, socialize as much as I can, force myself to, even though I’m feeling like, maybe socially inept some days from coding so much. I still force myself to go out. Another thing I like to do is, a good regimen of certain [neutrogroups 00:20:13].
So are any of you guys familiar with neutrogroups? So, like caffeine, [inaudible 00:20:24], [inaudible 00:20:26], a whole slew of stuff. Really good for focusing but if you take it for too long you get really joyless, for a novice like me, so you have to take breaks every now and then. I like to take breaks and they you get the wicked bad, like, caffeine jolt, headaches, all that kind of stuff, so anyway …
In terms of productivity tools, I use a tool called RescueTime that’s really awesome. It’s a little bit creepy because they ask you to, like, they say they’re not going to sell your data but then they still track everything and their like, “Oh, we’re not going to sell it, tell us what you’re doing when you’re not at your computer.” It’s like, ah, no thanks. So I use that in combination with something called Stayfocused and then I don’t use social media at home. I just don’t use it. I don’t really have social media but I still don’t use it at home because it’s just too tempting. I also use Spacemacs. I don’t know if you’re familiar with Spacemacs for [inaudible 00:21:17]. It’s really awesome. It’s basically Emacs within … so it’s Emacs with Evernotes … super awesome, it keeps you super productive.
In conclusion, be sure to brush your teeth. Praise the sun … vitamin d is really good, go outside and get exercise. Now, as I say that in a basement underground. Okay, so Jeff, thanks for listening to me. Do you have any questions or suggestions?
