Categories
async python

Background tasks on the cheap

When integrating with third party API’s you need to make sure that your requests reach the third party. In case of issues on their end you want to retry and best not to interrupt the flow of your application or even worse pass the information about such issues to the end user (like leaking 503 errors).

Most popular solution is to use a background task and there are tools for helping with that: celery, python-rq, or dramatiq. They do the job of executing the code in the background but they require some extra infrastructure to make it work, plus all the dependencies they are bringing in. I have used them all in the past with great success but most recently decided to write a basic background task myself.

Why? As I mentioned earlier all of them require extra infrastructure in a form of a broker that most of the time is redis, this implies changes to deployment, requires additional resources, makes the stack more complex. The scope of what I had to do just did not justify bringing in this whole baggage. I needed to retry calls to AWS Glue service in case we maxed out capacity. Since the Glue job we are executing can take a couple minutes our calls to AWS Glue had to be pushed into the background. I’ll give you the code and summarize what it does. By no means this code is perfect but it works 🙂

# background.py
import threading
from queue import Queue

task_queue = Queue()
worker_thread = None


def enqueue_task(task):
    task_queue.put_nowait(task)

    global worker_thread
    if not worker_thread:
        worker_thread = _run_worker_thread()


def _process_tasks(task_queue):
    while task_queue.qsize():
        task = task_queue.get()
        try:
            print(f"Do stuff with task: {task}")
        except Exception as e:
            task_queue.put(task)

    global worker_thread
    worker_thread = None


def _run_worker_thread():
    t = threading.Thread(target=_process_tasks, args=(task_queue,))
    t.start()
    return t

Public interface of this small background module is one function enqueue_task. When called task is put on the queue and thread is started. Each subsequent call will enqueue task and thread will be closed after it processed all of them.

I find this simple and flexible enough to handle communication with flaky services or services with usage caps. Since this can not be scaled it has limited usage, but HTTP calls are just fine. This code had been inspired by one of the talks of Raymond Hettinger regarding concurrency and queue module.

Processing…
Success! You're on the list.
Categories
typescript

Using classes to differentiate types in TypeScript

I like types and I like TypeScript, mostly because of the types. Even though it does not go as far as I’d like. After going to one interesting talk regarding safe domain in the code base I have an idea to give it a go in TypeScript. This is going to be a short journey in possible options I had to use types to make my code a bit safer.

I have a piece of code that integrates with two APIs, Bitbucket and Jira, and as usual it uses tokens to do it. The idea is to define a type describing token that would not be mixed up. The compiler would tell me if I made a mistake and passed Jira token into function that expects one from Bitbcuket. Tokens are just strings so thefirst option is type alias.

Type alias

So I had defined two type aliases, one for each API, and then a function that would only accept one of them. If you read TypeScipt documentation on types you know that this would not work.

Aliasing doesn’t actually create a new type – it creates a new name to refer to that type. Aliasing a primitive is not terribly useful, though it can be used as a form of documentation.

The below code will compile and according to tsc there is nothing wrong here. Here is a link to code in TypeScript playground.

function runAlias(a: BitbucketToken) {
    return a;
}

type BitbucketToken = string;
type JiraToken = string;

runAlias("a" as JiraToken);
runAlias("a" as BitbucketToken);

Interface

My second thought was to try and use interface but it was dissapointing as well. TypeScript uses what is called "structural subtyping” and since token types have similar sctructure they were identified as compatible but that was not my goal. Here is a link to code in TypeScript playground

interface BitbucketToken {
    value: string;
}
interface JiraToken {
    value: string;
}

function runInterface(a: BitbucketToken) {
    return a.value;
}

runInterface({ value: "a" } as BitbucketToken)
runInterface({ value: "a" } as JiraToken)

Class

Next in line is class and as you can see boiler plate ramps up. Result is unfortunately same as with inteface version. It should not be a sruprise to me as documentation clearly says what is going on.

TypeScript is a structural type system. When we compare two different types, regardless of where they came from, if the types of all members are compatible, then we say the types themselves are compatible.

Here is a link to code in TypeScript playground

// class version
class BitbucketToken {
    value: string;
    constructor(value: string) {
        this.value = value;
    }
}

class JiraToken {
    value: string;
    constructor(value: string) {
        this.value = value;
    }
}

function runClass(a: BitbucketToken) { }

runClass(new BitbucketToken("a"))
runClass(new JiraToken("a"))

Class with private or protected

Last and final option, as it did the job, was class but with private or protected property. Again documentation helps with understanding why it works.

TypeScript is a structural type system. When we compare two different types, regardless of where they came from, if the types of all members are compatible, then we say the types themselves are compatible. However, when comparing types that have private and protected members, we treat these types differently. For two types to be considered compatible, if one of them has a private member, then the other must have a private member that originated in the same declaration. The same applies to protected members.

This version finally worked and tsc complained when tokens where mixed up so I went with it in my personal project. Both options work private or protected.

Here is a link to code in TypeScript playground.

// class version
class BitbucketToken {
    private value: string;
    constructor(value: string) {
        this.value = value;
    }
}

class JiraToken {
    private value: string;
    constructor(value: string) {
        this.value = value;
    }
}

function runClass(a: BitbucketToken) { }

runClass(new BitbucketToken("a"))
runClass(new JiraToken("a"))

Processing…
Success! You're on the list.