Flask-Split

Flask-Split is a Flask extension for A/B testing your web application. It is a port of Andrew Nesbitt’s excellent Split A/B testing framework to Python and Flask.

Installation

The easiest way to install Flask-Split is with pip:

pip install Flask-Split

You will also need Redis as Flask-Split uses it as a datastore. Flask-Split only supports Redis 2.0 or greater.

In case you are on OS X, the easiest way to install Redis is with Homebrew:

brew install redis

If you are on Ubuntu or other Debian-based Linux, you can install Redis with APT:

sudo apt-get install redis-server

Quickstart

In order to start using Flask-Split, you need to first register the Flask-Split blueprint to your Flask application:

from flask import Flask
from flask.ext.split import split

app = Flask(__name__)
app.register_blueprint(split)

After that you can start A/B testing your application.

Defining an A/B test

You can define experiments with the ab_test() function in a view or a template. For example, in a template you can define an experiment like so:

<button type="submit">
  {{ ab_test('signup_btn_text', 'Register', 'Sign up') }}
</button>

This will set up a new experiment called signup_btn_text with two alternatives: Register and Sign up. The first alternative is the control. It should be the original text that was already on the page and the text you test new alternative against. You should not add only new alternatives as then you won’t be able to tell if you have improved over the original or not.

Tracking conversions

To measure how the alternative has imcpacted the conversion rate of your experiment you need to mark a visitor reaching the conversion point. You can do this with the finished() function:

finished('signup_btn_text')

You should place this in a view, for example after a user has completed the sign up process.

Configuration

The following configuration values exist for Flask-Split. Flask-Split loads these values from your main Flask config which can be populated in various ways.

A list of configuration keys currently understood by the extension:

REDIS_URL
The database URL that should be used for the Redis connection. Defaults to 'redis://localhost:6379'.
SPLIT_ALLOW_MULTIPLE_EXPERIMENTS

If set to True Flask-Split will allow users to participate in multiple experiments.

If set to False Flask-Split will avoid users participating in multiple experiments at once. This means you are less likely to skew results by adding in more variation to your tests.

Defaults to False.

SPLIT_IGNORE_IP_ADDRESSES

Specifies a list of IP addresses to ignore visits from. You may wish to use this to prevent yourself or people from your office from skewing the results.

Defaults to [], i.e. no IP addresses are ignored by default.

SPLIT_ROBOT_REGEX

Flask-Split ignores visitors that appear to be robots or spider in order to avoid them from skeweing any results. Flask-Split detects robots and spiders by comparing the user agent of each request with the regular expression in this setting.

Defaults to:

r"""
(?:i)\b(
    Baidu|
    Gigabot|
    Googlebot|
    libwww-perl|
    lwp-trivial|
    msnbot|
    SiteUptime|
    Slurp|
    WordPress|
    ZIBB|
    ZyBorg
)\b
"""
SPLIT_DB_FAILOVER

If set to True Flask-Split will not let ab_test() or finished() to crash in case of a Redis connection error. In that case ab_test() always delivers the first alternative i.e. the control.

Defaults to True.

Web Interface

Flask-Split comes with a web frontend to get an overview of how your experiments are doing. You can find the web interface from the address /split/.

If you would like to restrict the access to the web interface, you can take advantage of blueprint’s hooks:

from flask import abort
from flask.ext.split import split

@split.before_request
def require_login():
    if not user_logged_in():
        abort(401)

API reference

This part of the documentation covers all the public classes and functions in Flask-Split.

flask.ext.split.ab_test(experiment_name, *alternatives)

Start a new A/B test.

Returns one of the alternatives. If the user has already seen the test, they will get the same alternative as before.

Parameters:
  • experiment_name – Name of the experiment. You should never use the same experiment name to refer to a second experiment.
  • alternatives – A list of alternatives. Each item can be either a string or a two-tuple of the form (alternative name, weight). By default each alternative has the weight of 1. The first alternative is the control. Every experiment must have at least two alternatives.
flask.ext.split.finished(experiment_name, reset=True)

Track a conversion.

Parameters:
  • experiment_name – Name of the experiment.
  • reset – If set to True current user’s session is reset so that they may start the test again in the future. If set to False the user will always see the alternative they started with. Defaults to True.

Changelog

Here you can see the full list of changes between each Flask-Split release.

0.2.0 (2012-06-03)

  • Added REDIS_URL configuration variable for configuring the Redis connection.
  • Fixed properly finished() incrementing alternative’s completion count multiple times, when the test is not reset after it has been finished. The fix for this issue in the previous release did not work, when the version of the test was greater than 0.

0.1.3 (2012-05-30)

  • Fixed finished() incrementing alternative’s completion count multiple times, when the test is not reset after it has been finished.

0.1.2 (2012-03-15)

  • Fixed default value for SPLIT_DB_FAILOVER not being set.

0.1.1 (2012-03-12)

  • Fixed user’s participation to an experiment not clearing out from their session, when experiment version was greater than 0.
  • Fixed ZeroDivisionError in altenative’s z-score calculation.
  • Fixed conversion rate difference to control rendering.
  • More sensible rounding of percentage values in the dashboard.
  • Added 90% confidence level.
  • Removed a debug print from Experiment.find_or_create().

0.1.0 (2012-03-11)

  • Initial public release

License

Copyright (c) 2012 Andrew Nesbitt

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Fork me on GitHub