Bot subscription process and Celery tasks

Thu 15 September 2016 by David Hontecillas

This is a draft of the flow process to subscribe to new BJJ events notifications that I want to have for

In order to check all the dialogs I prepared a unit test to fake the subscription process (with predefined user responses) so I can easily see how will it look when released in production.

subscription flow for the bot

(Ehem! yes, those event images are only for the test :)

You must take into account, that until your app is not approved for subscription messages, before you run a test like this, you must send a message to your page so your server can respond to you. Check the platform policy overview.

I'm writing this just some days after facebook released new additions to the bot, like the webview extensions with support for integrated payments that confirms that the approach of using facebook as simpler apps that can receive notifications is not a bad idea at all (specially for MVPs).

One cool addition that comes with that update is that quick replies now support sharing your locations. Currently I'm using a geocoder api call to transform addresses into longitude, latitude pairs, but I can benefit from this new button to know where the users are.

Also, we have a new share action. I will certainly implement this for the events. Those guys at facebook want to keep me busy!

Need for background tasks

Once users are subscribed, I need to send them notifications every time a new event in their range is added. I should not do it in the same process that is handling the request, because this is going to be too time consuming (look if the event is inside the radius of each subscribed user, and then post the messages).

So, I decided to add Celery to the stack!

The most common configurations are with RabbitMQ (the recommended one) or Redis as message brokers. I decided to go with the second one, since I will use Redis for other stuff. Also for the current size of the project is more than enough (and less infrastructure to maintain).

My process is:

  1. Manually setup it in the development environment (a Virtual Machine runnning in my laptop), that has the exact same configuration that will be on the production servers.

  2. Once everything is running smoothly, I modify my ansible playbook, and deploy it to the demo site. And when the time comes, I will play it into the production site.

So, these are the steps without too much detail:

Installing Redis

Although the official site recommends to build it from sources, I am going with the easy path of installing the one that comes with the official Debian distribution.

sudo apt-get install redis-server
sudo apt-get install redis-tools

Once installed, we can check that it is already started

sudo service redis-server status
redis-cli ping

pip install Celery

In order to use Celery and redis, we can do it with a pip bundle instead of installing each one of the python packages

pip install -U celery[redis]

I just followed the official celery with redis documentation for creating a celery worker process, setup a basic task that sends a facebook message to myself, and then bringing up the django shell to try to trigger the background task.

Development server ready

Once I had the develpment server working as expected, I updated the pip_packages files for develop and production to reflect the required changes. (The demo site uses the production settings, and I use demo as staging too).

Updating ansible scripts

So, at this point we need to install redis (that we were not using) and for that we create a new task file for ansible install_redis_db.yml that will be imported in the main.yml file.

- name: ensure redis packages are installed
  apt: name={{item}}
  sudo: yes
    - redis-server
    - redis-tools

- name: check redis server is running
  service: name=redis-server state=started enabled=yes
  sudo: yes

To have celery workers running in background and ready to receive tasks have several options to deamonize celery.

In my case, I am going with the supervisord option, because I am already serving the django app using gunicorn and supervisord.

Following the official documentation, you can reach the sample config file for celeryd, but if you prefer, there is a simpler example in the Thomas Sileo blog post about celery and supervisord

My setting looks very much like Thomas Sileo's one, with the only addition of an environment variable that points to my current deployment dir, so it can find the django packages.

command={{virtualenv_dir}}/bin/celery worker --app=bgtasks -l info

Run the playbook into the demo site and test

With a simple:

ansible-playbook -i demo_hosts -K django_server_setup.yaml

the deploymnet is done.

I make a live test: I've put a simple message notifying myself (with my known facebook receiver_id hardcoded) when any user reports a new event.

So I report an event in the demo site, and ... it's there, the message poping in my facebook page ! :)

Of course this test will be removed and notifications implemented properly.