Archive

Archive for the ‘Development’ Category

Python Test Spy Engine – MI7

Introduction

MI7 is my new pet project. I grew tired of the other mocking/stubbing/spying engines in Python that never did quite what I expected them to.

I’ve been TDDing for a while now in .Net, Ruby, Python and JavaScript. I’ve got my fair share of experience, so I figured I’d give my 2¢ in this issue.

You can check the project at https://github.com/heynemann/mi7/wiki. It’s got a nice tutorial and is currently in 0.1.1 alpha release.

I’ve got a lot of work to do to make it my main tool for test support, but I’m going to get there. Without further delay, let’s get to it.

Why another Spying Engine?

If you check the MI7 wiki you’ll see that I don’t have anything against any single python test support engine. I just haven’t found one that suits my needs and those needs ONLY. IMHO they all do too much. I want a simple, straightforward, fun to use spying engine.

Don’t get me wrong, but I do not believe in mocking in Python. Or stubbing for that matter. Both are akin to dependency injection, IMHO. It just isn’t pythonic.

Python has been around for a while. In this time, there has been a certain Modus Operandi of doing work. This MO has never included injecting your dependencies around. I figured that’s why I feel the weirdness on the part of the mocking/stubbing/spying tools.

With MI7 I’m trying to interfere as less as possible with your code. Production code should be optimized to be production code, and not changed to accommodate your poor testing tools. In the Ruby community they try HARD to make tests and code as clear as possible, not make code work according to tools. I’m trying to get some of that.

The last point of me doing MI7 is to have some fun, and I’m trying to bake in the library as much of that fun as possible, with the spy agency metaphor. Hope you enjoy as much as I am.

Test Sample

Ok, so I’ll write a test with MI7. It’s pretty simple:

from controllers import MyController
from models import User
@new_mission
@agent.spy(User)
def test_user_is_authenticated():
    agents.User.intercept('is_authenticated') \
               .returns(True)
    agents.User.intercept('username') \
               .as_attribute('Bernardo')
    ctrl = MyController()
    result = ctrl.index()
    assert result == "Welcome Bernardo"

So what’s happening here. I’m telling MI7 to keep an eye in the User model, wherever it may be used. Then I’m instructing the User agent (the agents get their code-name from the target they are spying), to intercept calls to is_authenticated and username and return my values.

Now the controller code:

from models import User
class Controller(object):
    def index(self):
        user = User()
        if user.is_authenticated():
            return "Welcome %s" % user.username
        return "Unauthorized"

As you can see, there's not a single line of code in that controller that says "I'm testable". It's just plain old python coding.

Current Status

Currently MI7 supports intercepting modules and classes and telling agents to intercept methods and attributes and to raise exceptions.

Impersonation (stubbing) may come next. Definitely assertions are coming, like what an agent has seen and such.

Conclusion

I’ll keep going with MI7 development as much as I can, because I believe the Python community needs better testing tools and I’m willing to put extra effort into this.

Dream Team – Part III – Processes

The Value

As explained in the previous post, the first value outlined in the iNews team is that processes serve the sole purpose of being challenged and improved.

The Discussion

The team gathered at a meeting room. At the meeting there were John Miller, Susan Lawrence, Jane Collins, Jake Preston, Christian Fields, Joseph Ross and me.

I was invited just to listen, no talking allowed. I was in charge of time keeping so they would respect the time box for the meeting (the only thing I was allowed to say).

The first one to speak was John:

John: Hi guys! It’s a pleasure to be working with such a distinct and diverse team.

We have a very ambitious and interesting project ahead of us. We are in charge of changing the way people think about mobile news.

But before we set out to do just that, I’d like to discuss with you our team values.

All the other team members looked puzzle. Susan quickly replied:

Susan: I don’t get your meaning, John. I thought we were supposed to use scrum like the rest of the company.

If we are to use it as a process then we already know our values: the twelve principles in the agile manifesto.

What exactly do you mean by “our values”?

Every pair of eyes in the room turned to John. Tension in the air (ok, that’s just me being dramatic):

John: Very well observed. The issue here is that those are the agile manifesto principles. Not OUR principles.

We might end up with exactly the same set of principles, but then they’ll be our principles as well.

If we get to that, I’m sure we’ll live by those and in every decision you’ll abide by them.

A sense of shared understanding filled the room. “That makes just so much sense”, I thought.

Jane still looked uneasy, though. John, as a good leader sensed that and asked her what was worrying her.

Jane: Well, as a designer I’m not as used as you guys to a formal process. I’m very used to change, though.

It’s very clear to me that change is a competitive advantage.

John: I see, Jane. It is a good thing that you mentioned the process.

I really believe that processes are guides to help us interact and they serve no other purpose than to be challenged and replaced.

Christian: Coming from an open-source background, I’m very used to challenging and replacing “processes” of all kinds: contribution, releases, licensing.

Usually you get the processes from some other well-known successful project and start adapting them to your project.

Joseph: I like that discussion a lot. I come from a ScrumMaster position in the company from the early days of scrum.

A lot of failed Scrum implementations come from the fact that people get the practices without understanding the principles.

This leads to process paralysis. They won’t change anything in the process even if it gets in their way just because some book says they *HAVE* to do it this way.

John: What do you think now, Susan?

Susan: I couldn’t agree more with you guys.

I never thought that people failed to understand that processes are mutating things.

I always tried to adapt processes to fill my needs.

Jake: Even though I come from a very different background, what you said makes perfect sense.

I do share the concern with Jane, though. What if the current process does not really include us in the loop?

John: That’s exactly why we are suggesting that the process must be challenged and replaced with a better fit.

If ANYONE in the team feels some practice is not worth doing and have ANY better idea we should at least give it a try.

Best case scenario, we get best at doing what we have to do. Worst case, we learn. Looks like a win-win situation to me.

At this point I’m just amazed at how easily people with such different backgrounds connected over an ideal of delivering more.

Joseph: Well, I can confidently say that we have one of our values defined.

Processes are guides that should and will be challenged any time anyone thinks of a better way of doing some practice (or even not doing it at all).

There are no fool ideas or experiments. There are no reprimands to suggesting improvements by anyone in the team.

Does that sum it in an understandable way?

Everyone nods in agreement.

John: Ok, so we abide by this value at all times until we find it to be a poor fit to our team and change it. Ok?

Again happy nods in agreement.

Me: Awesome! Done with 5 mins to spare!

Susan: Yeah, and since I’m hungry, what do you guys say of a trip to Starbucks?

All: YAY!

Hmmm… Caffeine addicted, are we?!

They go happily to Starbucks and then come back to discuss next value.

Conclusion

This value may seem counter-intuitive at first. If I keep challenging a process that means it’s not good enough. What’s the point of having a process if it’s not good enough?

The team’s answer to that is that the process is just the current best practices for delivering value and as there’s no such thing as THE best way to deliver software, they are supposed to be continuously challenged, experimented with, improved and replaced.

They do not believe that any of them know exactly the best way of doing anything, yet they uphold the ideal that together, through iterative refinement they can keep improving the way they do things.

This is the reason for this value. The process is not what’s important. Delivering business value is. The process is just the means to do that.

Dream Team – Part II – The Team’s Values

Introduction

In the last part of the ACME’s iNews saga, John, Susan, Jane, Jake, Christian and Joseph set out to start the product.

They decided that, in order to consistently deliver and provide value to the company, they needed to get their team’s values straight. What do they believed in as a group? What principles and ideals would they uphold and work by? So they got together and after some brainstorming they came to the following Team Values Statement.

Team Values

The following values are what the entire team considers fundamental and is committed to upholding.

  1. Every process exists for the sole purpose of being challenged and improved;
  2. The best time to solve any problem or defect is now;
  3. Code only has value in production. Unreleased code is waste;
  4. Respect and Fun, with responsibility;
  5. We own the product!;
  6. Automate everything you possibly can, except people;
  7. Tranparency above everything. No questions can be invalid or taboo.

Mission Statement

They finish their value statement with their mission statement:

  • These are our values and we commit to them and to one another.
  • We also recognize that our purpose is to create value for the organization.

Conclusion

These values are not something the team “agreed” on, or just accepted. This is a set of common values that they share and will uphold above everything.

There’s a big difference in agreeing with or accepting something and sharing a value with someone. The fundamental difference is commitment. People are committed to their values and this team is committed to the values above.

I’ll explain in detail in further posts every single one of the values they outlined as they discussed (I was invited to the discussion).

Dream Team – Part I – The People

Introduction

Tonight I was wondering if I could describe a dream team. I’m not talking about dream team members, since I’ve only worked with very smart and committed people for this last year and a half in globo.com. I’m talking about a fictitious team where people would apply everything I believe a dream team would: profound knowledge, scientific method for problem solving, set-based design, continuous improvement, continuous deployment, iterative discovery, design and development, and some other things.

As is very well put by Mary and Tom Poppendieck in their book, it’s never the workers faults that create bugs, delivery rescheduling or customer dissatisfaction. It’s ALWAYS the system. So the main goal for this team is to make the system as mistake-proof as they possibly can, and then some. Through these posts I’ll try to describe the people, the process and its improvements and a product they are developing.

The Product

The team will be doing an iPhone/iPad app to deliver news to clients of a news agency. It’s creatively named iNews.

The Team

Even though this is a fictional team, I want to describe them (even with pictures – all Creative Commons) so I can translate that PEOPLE are the goal, the main thing. These are the people set to do a FANTASTIC product for my fictitious company, ACME Software.

http://www.flickr.com/photos/ian_munroe/4174136961/ John Miller

John Miller is the tech lead for iNews. He used to work for a very large news agency and has more than 10 years of experience developing software.

John has a lot of experience with agile practices and lean software development. ACME’s board of directors expect him to lead the team to deliver a surprising, efficient and competitive product.

John has experience with static and dynamic languages, yet no experience with iPhone development. He’s eager to learn all about it, though. He wonders what kinds of architectural issues the iPhone/iPad development model poses.

 

http://www.flickr.com/photos/lara604/2369412952/ Susan Lawrence

Susan Lawrence is a senior engineer with the company. She is very knowledgeable of Scrum and in previous projects for ACME she loved the methodology. She feels that something was missing, though. There were some issues identified by the team and yet the team did little to solve them.

She has no experience with mobile development but is very eager to learn as much as she can. User experience is a subject that she cares deeply and is looking forward to working with Jane Collins.

She is the author of Stinks, the Continuous Integration server being used by some teams at ACME, so you can tell that she cares a lot about this practice.

 

http://www.flickr.com/photos/dichohecho/2556568314/ Jane Collins

Jane Collins is the user experience designer for iNews. She has a success track record with the company on several previous products.

She used to decide on  her own what the user experience was supposed to be. This has proven to be the ineffective way of doing this, since the team always had issues implementing what she designed.

In her last project she tried a more tight integration with developers and they provided invaluable insight into what the users might value and what they wouldn’t.

She’s looking forward to the challenges of designing a consistent user experience for such distinct devices as the iPhone and iPad.

 

 

Jake Prestonhttp://www.flickr.com/photos/titlap/3936883765/

Jake Preston used to be an expert at front-end software development. Tired of this separation of front-end and back-end developers, he decided he would be as knowledgeable on back-end development as everyone else in the company (if not more).

Jake has been studying software engineering practices in general and learning more in every project he´s in. The iNews product has several interesting challenges for him. Among them, the different presentation requirements and capabilities of the devices.

He’s very concerned with how can interface automated testing be done in such devices, as well.

 

 

 

http://www.flickr.com/photos/greggoconnell/252976245/Christian Fields

Christian Fields has just joined ACME. He never worked with news agencies, but has a very good track record with open source projects, being a contributor to large and very large projects.

He has a very strong culture of sharing and contributing. He expects to be able to apply this knowledge to iNews, since collaboration with customers and to some other teams will not only be required, but key to success.

He values second to none automated testing and versioning, being so used to rejecting patches that do not provide automated tests and releasing early and often to gather feedback of the community.

He is VERY excited to be working with a team of smart people in a very promising product.

 

http://www.flickr.com/photos/flechtnerby/4817094778/Joseph Ross

Joseph Ross is the project manager. His role is to remove any impediments that stop the team from fulfilling the values in their value statement (next part).

He has a very strong process background being a certified scrum master and PMP. He thinks methodologies are guides whose only purpose is to be improved and replaced with the new improved process.

He has been in the company for more than 10 years, thus he knows virtually every employee and knows exactly how ACME and its people operate, as well as what the company’s values are.

Conclusion

These are the people who, together, will succeed or fail in delivering an innovative news content delivery application for the iPhone/iPad platform.

More about what their values are in the next post.

Django Compressor – Minify/Reduce Requests

The Problem

Nowadays with all the JS frameworks and CSS Frameworks, our web applications end up with a very rich and interactive UI (also called User Experience). With the easy-of-use that this tools bring, many plug-ins and third-party code gets integrated in our apps. Usually this code comes bundled with many CSS and JS files (usually one for each plug-in). They easily pile-up to become a bottleneck of browser rendering performance.

As Yahoo points out in this article, one of the best ways to speed up your website is to drastically reduce the number of requests that the user’s browser has to do.

Another way to improve performance is to minify your CSS and JS. The minify action reduces the number of bytes that go through the wire, thus improving download speed for your users.

At the company I work for we faced a big issue with our pluggable app. After many plug-ins and customizations by clients, our pages were REALLY big with MANY (100+) requests. You can easily guess that this is not a situation you want your website to be in, right?

Django Compressor

Enter Django Compressor, the pluggable app for Django that provides a template tag to compress CSS and JS. It does more than that, though. Django Compressor provides an extensible architecture that allows you to implement your own template parser, your own template filters and your own static CSS and JS Storage. We’ll discuss more of each in the following topics.

I just want to thank beforehand all the effort that has been put in this project and all the involved people. The quality of the project has really surprised (a good surprise) me and I’m really glad to have found it (via Gabriel Falcão).

Basic Usage

As you can read in the readme file (or in github main page), it’s really simple to set Django Compressor up. The steps are:

  1. Add ‘compressor’ to INSTALLED_APPS in the settings.py file for your Django app (make sure compressor can be imported using ‘import compressor’ in python).
  2. You don’t *HAVE* to, but I like to specify whether I want compression on or off (the default is the opposite of DEBUG, meaning that in DEBUG mode you don’t compress and with DEBUG off you get compression). If you, like me, want to specify whether to compress or not, just add COMPRESS=True to your settings.py file (anywhere).
  3. Go to a template where you have many CSS and include the following:
    1. {% load compress %} before using any compression.
      This loads the template tag.
    2. In your templates, you compress your stylesheets:
      {% compress css %}
      <link type="text/css" rel="stylesheet"
            media="screen"  href="/media/my.css" />
      <!-- many more css files -->
      {% endcompress %}

That’s all there’s to it. After reloading your view you should see that the my.css link got replaced with <<some_hash>>.css. The hash here is generated based on the contents that got compressed. This means that all the CSS requests that were being done previously will now be done in one go. The same goes for JS, the only difference being that you supply ‘js’ as argument to compress, like this:

{% compress js %}
<script type="text/javascript" src="/media/my.js" />
<!-- many more js files -->
{% endcompress %}

It gets better, though!

Compressor Filters

Django Compressor uses a really simple and easy to understand filter architecture. Basically whatever filters you specify for JS (in the COMPRESS_JS_FILTERS setting) and for CSS (in the COMPRESS_CSS_FILTERS setting) will get called sequentially in order to modify the output.

What this means is that if you want to validate, minify and then concatenate your CSS files, all you have to do is specify the proper filters (or implement your own).

In our current scenario we are using the following settings:

COMPRESS_CSS_FILTERS = [
     'compressor.filters.cssmin.CSSMinFilter'
]
COMPRESS_JS_FILTERS = [
     'compressor.filters.jsmin.JSMinFilter'
]

As you can see I’m only using the filters that are already provided with Django Compressor. What this means is that I get for free CSS and JS minification.

If you want to implement your own filters I suggest you take a look at the code for the current filters in Django Compressor (in the filters folder in the source code).

Django Compressor FileStorage

Django Compressor uses Django’s concept of a File Storage. A file storage is an entity responsible for every related file operation that Compressor needs to execute (verifying if a file exists, building file paths, opening and saving files, etc.).

Here at our team we have media distributed in many apps, all of them inside a /media folder. We distribute media like that because our clients plug-in only the apps that they need, so it makes sense that the medias stay where they actually belong. That works pretty well with the way django loads media but deployment is a little harder. We had to work our own deployment scheme (subject for a later post).

By default django compressor only uses the projects media folder (as specified in the MEDIA_PATH setting). We had to change it so we could use it with our apps. It was as easy as implementing our own File Storage class. You can check it at my fork in http://github.com/heynemann/django_compressor. As of the time of this post our code has not been integrated to the Django Compressor project. We are hoping that it does soon.

To use our custom storage all we had to do was specify it in the settings.py file with:

COMPRESS_STORAGE = 'compressor.storage.AppSavvyCompressorFileStorage'

And that’s it! Now the pages that refer to medias in our apps also get compressed.

Conclusion

Django Compressor is a really cool project and very worth contributing to. It already features many filters, parsers and now two storages. If you want to speed-up the loading time of your pages and you are using Django, you definitely should consider it.

Follow

Get every new post delivered to your Inbox.