DevSummit07:Agile Development and Rapid Prototyping: What Works in the Nonprofit Context
Facilitated by Emily Tucker, Grameen Foundation, Kevin Smith, Benetech, and William Pietri, Scissor
New methods for prototyping and developing software are revolutionizing the ways tools are built for nonprofit applications. Practitioners of these techniques will share their experiences and demo the processes.
Many nonprofit software development efforts follow traditional "waterfall" development approaches. This session will explain why project managers and "requirements drivers" should consider embracing agile and other new methodologies, and why, in particular, such approaches are a good fit for Open Source projects that leverage volunteer resources.
Addressed in the discussion will be the initial hesitancy that NGOs (or any org, for that matter) may have when making the change, and specifics of making the transition from a waterfall to agile approaches will be described. Specific recommendations for developers wanting to implement "best practices" in agile/extreme programming will be included.
This session will be run in an agile way: Folks in the room will be asked to suggest discussion topics, and then will rank them by interest level to determine which topics get covered in more depth.
Basic overview of methods
- Waterfall method - been around for decades, assembly line process, requirements, analysis, design, coding, supposed to be sequential process, but reality is that it isn't like that. At each stage, we discover changes that need to be made in earlier stages. But changing earlier work is expensive because it was heavily documented and signed off
- extreme programming - a specific form of agile software development - 2 people at the same time, no design, test all the time, radical ideas
- scrum - another agile methodology, small-team focused iterative process, less contrained than extereme programming
- Agile is responsive, iterative, not getting bogged down, not cowboy coding, requires discipline, there is no single checklist for agile - to be agile is to work with an agile mindset
- in waterfall, all requirements are decided up-front. Signed and sealed, no going back
- with agile, requirements are gathered in detail for the next week (or month) of coding -> prioritize those, then work from the top. At end, what's left undone are the lowest priority (least important) features
Polling - raise of hands
- Any experience with agile programming? About half the folks
- What size development teams do you typically work wiht? Almost all were 1-5 people
- Based on what you already know, do you feel positively towards agile? Mostly positive, few neutral, none negative (surprisingly)
- First, we had chaos in software development. Messy, not productive, poor quality (60's, 70's 80's). Waterfall was a response to this. It added valuable process, but became (in many cases) over-process, with 500 page requirements doc, etc...lots of ceremony
- programmers said - not fun, not effective, doing lots of things not related to what the customer actually needs
- agile is people over process - outcomes not the way that it happened. The agile manifesto lays out core principles
- [Audience member]: Told story of how he heard that agile came about - at the end of projects, teams tended to do agile practices out of necessity. They noticed how effective the project became and made this into a process for the entire project lifecycle.
Feedback loops -> the faster it happens, the better, faster you learn
- Years History/Board - mission statement
- Quarters - Public releases, budgets
- Months - Release plans, feedback from users
- Weeks - Team meetings, internal builds and releases
- Days - Pair programming
- Hours - Code and unit testing
We're using an Agile methodology to develop list of topics for the session
- Learn agile techniques and methodologies
and Different develpoment models
- How to integrate into dev process day-to-day
- How to do agile with distributed developers
- Non-programming agile
- How to sell agile to clients, management and developers
- Benefits for nonprofits
- How to include users
- How to ensure client can provide input (every 2 weeks)
- Are there types of technologies and projects that agile is particularly efficient of non-efficient
Ranking was done by a show of hands on each topic. Topic with the most votes was discussed first
1) How to integrate into dev process day-to-day
Example - "Blue cards". Blue cards are index cards used by some agile/extreme teams to track "stories", which are features or issues the team will work on.
Here is an outline of the "Planning Game" that many agile teams use to prioritize the stories:
- planning process -> number of things/features can go on forever - if they knew the costs, they would change what they wanted
- each feature goes on an index card - for the stack, put them in priority order
- as developers, estimate cost for each, then have client rank again based on value vs. cost
- 1 week iterations, stack - top items, developer does (for example) 3 of them
- the next week, client prioritizes again
- cards can also be added, and time estimate for any particular card can be changed, and new cards can get generated (but generally not during an iteration)
Question: What happens if the team only paid attention to the current stories, and a future story ends up requiring a technical change from what was implemented initially?
- for change -> client will have to pay for cost to change, BUT that cost tends to be much lower than you might think. Usually it costs about the same as it would have cost to do it that way originally. But very often, the initial simple inexpensive way turns out to be good enough forever, saving the extra costs.
- idea is to build what you need now
- modular code makes it easy to move/change/refactor (VERY important for agility)
- makes incremental changes easy - much easier than you would expect
- time estimates are done quickly - and developers get better with estimating
- each card can be do-able in one interation (1 week) so if they want "CRM", it gets broken down into cards/features
2) Comparing Different development models
(The audience requested that this topic be treated somewhat as a continuation of the first topic, since they are closely related)
- Agile separates responsibilities clearly: developers provide estimates, business sets priorities
- avoids imposing an impossible 'set of features with set schedule' on a team
- incremental, iterative, evolutionary design - martin fowler has an excellent essay online about evolutionary design
- design, code, design, code, in small increments (minutes, hours) rather than days or months of design followed by large amounts of coding
Question: When does code get cleaned up?
- ideally, cleanup (often in the form of "refactoring") happens during the normal the process. Occasionally, may have to set aside time just for internal technical work that the user won't see. Teams learn to build in time estimates for clean-up so it doesn't pile up.
- at the end of an interation, have something that works -> get user feedback early on. Literally start delivering installable executables within a week or two, even if it is as simple as just a login screen
- iteration lengths: 1 week (Kevin's preference), 2 weeks (very common) or one month (typical for scrum)
Question: What if client is not providing enough feedback? Should you use longer iterations?
- when issues with clients not giving feedback - shorten the iteration - to get them more mentally engaged (they see you more often) faster feedback loops
Question: what about client expectations? Are then disappointed with early iterations that hardly do anything?
- If they are used to seeing nothing for six months with traditional projects, they tend to be happy to see anything at all within the first month.
- As the project goes along, they start to focus on the improvements every week, and they build trust that you're able to deliver and that their money is going somewhere
- Expectations are important. Be sure the client knows what each iteration will include
- with some clients, agile can be done internally, but use waterfall with the client because it is what they know/expect/want/can handle
3) How to sell agile to clients, developers and management
- when advocating any change, learn the needs, fears, concerns of the audience
- from experience - flat fee projects - client relationship and language changes when using agile, proposals change, learn how to manage fear, etc...
- with fixed bid, fixed features, sometimes easier to do agile development internally, but use a waterfall-like presentation in the client management
how to sell to client
- point out reduced risk - for client, at any point, they can walk away and have something to show
An example of selling Agile
- for smaller projects - take 1 week - have the customer develop a rough list of requirements. Then, in 99% of cases, you can propose additional requirements that the customer recognizes as being really valuable. Point out that with waterfall, requirements end here. With agile, we can continue thinking of new cool ideas all the way throughout the process.
For any project, if at the end of the year-long cycle you deliver exactly what they wanted on day 1 - it will be considered a failure! By the end of the project, the market will have changed, and everyone on the team will better understand the domain and users, and will have thought of great new features. Change is inevitable...how will the project handle it? Will it embrace change (agile) or resist it (waterfall)?
If using waterfall with client and agile in development - can seem more flexible to client, can work with a fixed fee by adding to cost for changes, or dropping less needed requirements
Addenda from Kevin:
During the session introduction, I forgot to define "Rapid Prototyping", as compared to "Rapid Development". The key distinction is that most "prototypes" are quick and dirty throwaway code. Rapid development (as typically happens with agile methods) is very different from throwaway Rapid Prototyping. Different folks will use those terms differently, so beware of misunderstandings.
In our "advocacy" discussion, we focused on selling agile to external clients. Here is an essay I wrote about how to sell Extreme Programming to different audiences: http://qualitycode.com/html/Essay10.html My note at the end about Extreme being "hard to do" is true, but does not apply to agile in general. Agile methods like Scrum and Crystal Clear are pretty easy to adopt and execute, as long as the whole team is supportive of them. Of course, any organizational change can be a challenge.
Recommended Links and Reading Material:
http://qualitycode.com - Kevin's own site. Sadly, it hasn't been updated for years, but fortunately much of the material is timeless. All the "Agile" entries, and most of the Extreme/XP entries on the Links page are still active and valuable.
Sustainable Software Development (at half.com), Kevin's new favorite book about Agile development.
Martin Fowler's excellent article about agility: http://www.martinfowler.com/articles/newMethodology.html
Wikipedia articles about:
- Agile Software Development - Scrum, an agile methodology - Extreme Programming