Invoke

MC706.io
Python, Review, Tasks

For a long while, the python dependency that was holding me back from migrating entirely to python3 was fabric. For those who don’t know, fabric is a command line tool that allows you to define build and deploy tasks and execute them. It was designed to do some local work, then ssh into a remote host and execute a series of commands there to deploy a project.

The majority of what I ended up using fabric for was as a task runner. Need a short hand command for compiling sass? fab css. Want to sync a database with a remote, fab sync. Want to manage your changelog and release process with version numbers? fab release. I was using fabric the way I used gulp for javascript projects.

The problem there became I was forcing my hand in running python2.7 on a dependency that was for a build tool, not for the actual work product. When things like f strings, type hinting and asyncio becoming more of a thing in python3, I wanted to move. So I went looking for a replacement.

Luckily I didn’t have to look far. The fabric team was working on porting to python3, and the SSH module is what was holding them back. So they wrote invoke, the task runner side of the fabric as a standalone tool.

Basic Setup

The basic setup of invoke is a tasks.py file that looks a bit like this:

1
2
3
4
5
6
from invoke import task

@task
def test(ctx):
    """Runs the Test suite"""
    ctx.run('pytest tests')

You could then run this test task with either invoke test or inv test for short. The ctx argument is a reference to the context this command is being run in, so ctx.run() executes the command on the command line. The docstring for this task is used as the task description when you run inv --list: 

1
2
3
4
$ inv --list
Available tasks:

    test    Run the Test Suite

Building on that, you can add arguments to your wrapped function with defaults and accepted values, that allow for command line args to be easily added to your task. 

1
2
3
4
5
6
7
8
9
from invoke import task

@task
def test(ctx, smoke=False):
    """Run the Test suite"""
    if smoke:
        ctx.run('pytest tests -m smoke')
    else:
        ctx.run('pytest tests')

Here we added the ability for this test to be called like invoke test --smoke or for really shorthand: inv test -s.

Tasks can be pipelined at command time so a file that looks like: 

1
2
3
4
5
6
7
8
9
10
from invoke import task

@task
def clean(ctx):
    """clean up build directory"""
    ctx.run('rm -rf build')
    
@task build(ctx):
    """build the project"""
    ctx.run('python setup.py build')

Which can be pipelined as inv clean build.

Advanced Setup

As your tasks.py begins to grow, the pattern I follow is to break it out into a python package. This pattern looks like this: 

1
2
3
4
5
6
tasks/
    __init__.py
    test.py
    clean.py
    build.py
    deploy.py

Then then your __init__.py defining the task collection. 

1
2
3
4
5
6
7
8
9
10
11
12
13
# tasks/__init__.py
from invoke import Collection

from tasks.test import test
from tasks.clean import clean
from tasks.build import build
from tasks.deploy import deploy

task_collection = Collection()
task_collection.add_task(test)
task_collection.add_task(clean)
task_collection.add_task(build)
task_collection.add_task(deploy)

Now I have a clear separation of concerts between building, testing, deploying and all of the invoke code is neatly inside of the tasks package.

The next level of advanced is to abstract away the project specific parts of the tasks into context variables, and pass them in with the invoke.yaml file. This allows you to define common tasks across repos, copy the tasks directory from project to project (or include it in your cookiecutter), and configure your task runner using a config file.

For example, your clean task:

1
2
3
4
5
6
7
# tasks/clean.py
from invoke import task

@task
def clean(ctx):
    """Cleans previous build files"""
    ctx.run('rm -rf {BUILD_DIR}'.format(**ctx))

Then in the root of your project in an invoke.yml: 

1
BUILD_DIR: build

With this setup, I can have the same traveling tasks/ package that does things like manage my CHANGELOG.md and README.md files with the current version, run build and deploy commands, run testing, typechecking and linting, and manage git{hub,lab} issues. And all I have to do is copy over the invoke.yaml file and reconfigure it for each project.

Versus Make

The one reservation I have about invoke, is you have to have it installed in your virtualenv to start using it. Make, the other build tool I use, does not need to be installed at all on basically any environment (so long as you have no windows developers on your team), and can run arbitrary commands in a pipeline fashion.

Despite the much easier setup, I still use and recommend invoke simply because python is a million times easier to read and write than bash, especially inside of a python project.