Distributed Agile Development - an interview with Jason Mawdsley and Stephane Lussier

Typically, when people think of outsourcing development, especially offshore, they associate best practices with doing waterfall development – provide a big detailed specification up front that describes exactly what you want (otherwise known as BDUF - big design up front).

With proper controls in place, this may work for certain projects, especially more mature projects, but frankly it's hard to do BDUF for new product development. Many product development organizations are are embracing agile methods, because agile methods deal well with change, which is inevidable in NPD. They also provide a framework for making rapid forward progress while keeping focused on the highest value features.

I see a number of teams asking themselves, can we do agile development with a distributed team? Can I reap the benefits of agile when I have a team scattered around the globe?

So I decided to put together what I hope will become a series of interviews with people doing real-world product development, using agile methods with remote teams. Some say if you’re not in the same room, it’s nearly impossible to do agile development. Or if you can, you take a serious hit in productivity. I wanted to talk with people who were making it work.

For my first interview I started at home - with Jason Mawdsley, one of our Project Managers at Macadamian and a certified ScrumMaster, and Stephane Lussier, our VP of Engineering. At Macadamian, our team is spread out over three continents, but most of the work we do is new product development under intense deadlines, where BDUF isn't going to work.

Agile promotes a lot of communication, but normally in remote development you have to make do with “batch” communication. What are some strategies for closing the communication gap when you have teams scattered around the globe?

Jason: You have to set the tone for frequent communication, and have the tools in place - Skype and IM for example. We aim to have several live chats a week on a given project.
We have to be willing to adjust our times - someone from Ottawa may have to take the call at home first thing in the morning, or someone in Yerevan at the end fo the day. We also make use of a number of different technologies, like mailing lists and a Wiki (Confluence). We try to have a number of different tools to make communication easier, but everyone on the team needs to be on a common set of tools - the same IM, the same VoIP system

Stephane: You need to encourage people to ask questions. It's not easy, especially with newcomers. How do you do that? You tell people to ask questions. There are no stupid questions – this is not the time to shoot the messenger and tell them they should have known the answer.

How do you make the most of the one or two hour time overlap?
Jason: you need to mak sure that people in the west need to reserve some time their mornings for communication, and the in afternoon in the east

Stephane: But don’t wait to send questions until the overlapping hours – if something comes up in the afternoon and I need an answer, I'll send it, so that its at the top of the queue for when people come in in our Cluj and Yerevan labs. You only have a couple of hours of overlap, and somedays it may only be one – if it’s an urgent email, there’s good chance that people will act on it as soon as they come in

Tell me more about the mailing lists, what’s the benefit of having a mailing list for the project?

Stephane: It’s not the mailing list itself; it’s the philosophy of submitting code every day for review, in a place where everyone can see it. There are different ways of doing this – it doesn’t have to be a mailing list, but you want everyone on the team is showing activity every day, so that when people are going in the wrong direction you can realign them right away. It's the same concept as agile – everyone in the same room, talking, and aware of what everyone else is doing. Pushed to the extreme it’s similar to pair programming, but pair programming is a bit extreme for me, and not practical to do in a distributed team, but the benefit is the same - everyone reviewing everyone else's work.

Jason: I agree - it’s not the mailing list, the mailing list is just the tool. The culture is that we want everyone to be aware of all the questions and progress from the rest of the team

Stephane: We don’t want to do waterfall – big design up front by people close to the customer, and swing this offshore, and then hope for the best because you put enough detail in the spec. Some people make it work, but I think there are better ways

How important is it to have a local manager in each of your development labs? What qualities are you looking for?

Jason: No matter what kind of office you're opening, whether it’s a sales or development office, you need someone you can trust to oversee it. Someone who knows the culture, the rules, and the lay of the land.

Stephane: When there’s something that needs improving, you want to escalate it to someone close to the action. Solving probems remotely is more difficult than having a talk face to face with someone sitting in the same office. I get better results when say John, my manager in the Cluj lab, is sitting down with that person one-on-one to resolve an important issue.

You also want someone who’s persistent and patient. To develop an agile culture with new people takes time, especially if they’ve never worked in that mode. You want someone who will follow up on a weekly if not daily basis to reinforce the culture. -

How important is travel and getting face time?

Jason: It's important when you’re bootstrapping a project. When we sent one of our Project Leaders, Melissa, to our Armenia and Romania labs to kickoff a large project, it was the most effeicient way to get everyone up to speed. Also it helps build trust – you can get to know them on a social level.

Stephane: You build trust a lot faster. Once the trust is there, people are more open to discussing issues or even just tasks. They are more open and up front about things that need to be done. You want to have a strong trust relationship with the remote office manager- you rely on them to understand the culture, the process, and to coach people.

What types of people do you need to make distributed agile work? What qualities do you look for?

Stephane: that’s a good question… communication is important. Someone willing to communicate. You need people that understand what’s going on under the hood (of the OS, platform, etc). They have a logical brain, because in Agile development you don’t have a big spec with a lot of detail, so developers are asked to fill in the gaps; there are 2 ways to do this – be very bright, or communicate a lot and ask a lot of questions.

Jason, you’re a scrum master. Have we used all the Scrum concepts in distributed development? What pieces of Scrum work well and can be adapted, and what have you had to throw out?

Jason: We’re not using it full on. We’re using the most important tenets. We’re doing shortm 3-4 week sprints, we’re doing daily scrums but as a voice call, we have an infrastructure that lets people pick tasks, and a Wiki to track project info.

On many projects we let people pick their own tasks. We have Jira, which doesn't force you to assign tasks. You create them and can let people pick the ones they think they are best suited for … sometimes critical ones are assigned though, because you want them done by specific individuals.

Are there parts of Scrum that you say are impossible to do remotely?

Jason: No. It’s easier to do in person, but that doesn’t mean it’s impossible.

What can the open source community teach us about doing distributed agile development?

Stephane: A project can be successful and have high quality by people around the world, not just by people sitting in the same room. Open source projects don’t follow waterfall model – they are much closer to agile. It’s not true people need to be in the same room to develop using agile. Open source projects use mailing lists, or whatever forum, for sharing code. Typically the open-source projects that are working well have a very active mailing lists. There’s always someone to provide an answer and review work. They have a common forum to discuss issues. Also successful open-source projects have a few very good people running the show. They won’t let poor quality code slip by, they will set some rules about quality and enforce them.

Jason: I’d have to dig up the exact quote, but it goes something like “everyone is empowered to make their own decision”. In an open source project, everyone is volunteering – so everyone is passionate about it. So they pick the bugs and tasks they want to work on.

Stephane: Motivation is high and buy-in is really really high.

Jason: And because they contribute to the project by tasks, they don’t do big monolithic patches.
Stephane: Motivation and buy-in are at 100% - they do it in their free time. Who can say this in the business world?

Jason: Me? (laughs)

Stephane: So another thing we can learn from Open-Source is getting everyone's buy-in motiveates and gets good results.

How do you make distributed agile development predictable?

Stephane: Small patches. You need to be able track how you are doing in small doses. You also need to know your past performance. You need to have estimated tasks, then track how long they took, and adjust for next time. That’s the whole goal of doing Sprints – fixed time-boxes for iterations. One of the goals of this is to know how much you can do in one iteratrion. At one point, you know exactly what you can accomplish in 4 weeks. The same is true for a distributed team.

How do you do task allocation? Many agile methods advocate having team members pick the tasks they feel they are best suited for.

Stephane: We try to break things up in small 1-2 day tasks, but we'll also let the team do their own work breakdown. So when someone ends up with a task that is 3-5 days, they need to come back with a more granular plan. The developer has to do some breakdown . We eant people to think about how they wil tackle the problem, so that everyone knows the task is understood. This is a pattern that's hard to break. People are expecting, and this is everyone including North American developers, is that they will work a something for 2-3 weeks. I think it comes from school – you’re asked to go away and work on assignment for 3 weeks, and submit it when you're done. This doesn’t work, and you need to break this habit

Jason: In agile you want to favor communication over process and documentation, so you have to structure things to promote communication.

One of premises of agile is you have developers pick their own tasks. Does this work in a distributed team?

Jason: Yes but it's something you have to work on. You want to encourage them to pick their own tasks, but it's a culture that you have to instill. The advantage is they will pick a task that they are more motivated to work on.

Stephane: I find people that are just joining the project don’t want to just pick the task - they don't have the confidence yet. Normally you want to assign them something easy to get them some familiarity with the code base. Later, once they have confidence and experience they will be more apt to pick their own. This is true distributed or not.

Who plays the role of the user advocate in a distributed team?

Jason: The testers.

Stephane : I would kind of agree. Sometimes for us, it’s the real customer, who gets involved by being on the mailing lists, actively reviewing bugs and drops, and so on.

So how do you involve the actual customer?

Stephane: The customer has to want to be involved – and that’s the perfect scenario. You give them frequent drops, and they will give you feedback.

Jason: We don’t have the customer on-site, so we try to open everything up, securely, with Jira, Confulence, and SVN so they can see everything. We try to send frequent drops so they can provide constructive feedback. But you have to recognize that not every customer has this commitment – say it’s a non-core. feature. In this case it’s the Project Leader and Tester who have to take ownership of the product and act as the customer.

What kind of tools and systems can make distributed agile easier? What are the must haves?

Stephane: You need email (laughs). You need to be able to have phone conversations, so Skype is useful. You need live discussions. IM is a must. If you want to be able to distribute taks, you need some sort of task management system, or bug system that supports tasks like Jira. Video can be helpful sometimes.

One thing that is useful is working in a Virtual Machine– when you add a new developer, you give them a new VM – everyone has the same development environment.

Jason: it gives you a replicable environment – as the project gets bigger the setup is harder. It could take 2-3 days to set up a dev environment.

Stephane: you don’t want a situation where it stops working for one team member and no-one knows why, which happens frequently when you have a heterogeneous environment. You will lose time diagnosing, because troubleshooting will take more time. If the knowledge is with the team in India and the problem in Ottawa, you have to do multiple roundtrips – “send me your trace, etc.”

Jason: It can take four days to diagnose and solve in a situation like that .


What are some of the antipatterns that you need to avoid? Traps that you see managers who are new to distributed agile fall into?

Stephane: Not communicating expectations. In general, you need to over communicate. Just throwing a one-liner to someone in your lab halfway around th world, because you're in the habit of doing this with someone in the same office. You fire them an email - “hey I need this feature". What you forget is that when you do this, the guy next door to you comes and has a 30 min discussion about the feature. If you do this to someone remotely, and they don’t pick up the phone to talk about it, it won’t come back as you expected. So you need to provoke things – "hey, let’s have a 30 minute call tomorrow morning about this new feature you want to do."

The second is Monolithic patches – people going in the wrong direction for weeks at time, working on the same big feature, and you only learn it on a month later, and now it’s near the deadline and you don’t have time to recuperate.

Another is not having someone that plays the user-role – someone close the user of the project, You need to play it actively. Daily follow-up – are we going in the right direction? When I talk to people who say “it didn’t work out with my offshore team” – the common mistake I see is they write a big spec, swing it to the remote team, and we'll see you in two months; thinking that their involvement was done. If they had been involved throughout, I think they would have found the team ran into problems on week one.

But this is true for local teams as well – if the marketing teams specs out ome features and swings it to the dev team, and comes back in three months to see how it went, this won’t work.

Finally, not understanding that the cultures and work habits are a bit different. For example, sometimes with engineers in India, you'll find that things that for everything not mentioned it’s ok to make assumptions – you will be doing your job if you do everything that is said. It’s not customary to ask questions. In China, there’s a respect for authority to a degree you don't see in Eastern Europe or North America. Everytyhing that the manager says is true – you don't contest that. It’s not in the culuture to doubt, or double check that it is true. Most of the time the manager doesn’t have all the knowledge to make the low-level decisionsm, so if you say “use this library" - where with other cultures you assume you are making a suggestion, with a team in China you are making an order. The team will do what you say, even if it gets them very stuck. So not recognizing that each of your teams is different, and knowing how do work around those differences can make things difficult.

In closing, are there one or two things you'd like to leave people with - tips for new managers embarking on doing agile with a distributed team?

Stephane: To me, I would say be patient and persistent. In the beginning it will take a while – you won’t see the benefits right away, especially if you are changing the way you work.

Jason: Rome wasn’t built in a day. You need to take a mid-term view on this. You won't see immediate success. It will take training and culture changes to make it a success. You will see it manifest 2-3 months after it starts

Stephane: Encourage communication – both ways.

Jason: Open, transparent, quick, informal communication.

Stephane: And as a leader, you will need to enforce communication. But when you do Scrum, you enforce communication by default – there’s a Scrum meeting every day.