Django DDP Documentation

Release 0.17.0

Tyson Clugg

October 13, 2015

 Contents

1 Requirements 3

2 Scalability 5

3 Limitations 7

4 Installation 9

5 Example usage 11

6 Using django-ddp as a secondary DDP connection (RAPID DEVELOPMENT) 13

7 Using django-ddp as the primary DDP connection (RECOMMENDED) 15

8 Serving your Meteor applications from django-ddp 17

9 Adding API endpoints (server method definitions) 19

10 Authentication 21

11 Contributors 23

12 Installation: 25

13 Links 27

14 Contents 29
14.1 Change Log . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

15 Indices and tables 37

16 License 39

i
ii
 Django DDP Documentation, Release 0.17.0

Django/PostgreSQL implementation of the Meteor DDP service, allowing Meteor to subscribe to changes on Django
models. Released under the MIT license.

Contents 1
Django DDP Documentation, Release 0.17.0

2 Contents
 CHAPTER 1

Requirements

The core concept is that events are dispatched asynchronously to browsers using WebSockets and server-side using
a PostgreSQL extension to the SQL syntax called NOTIFY (and its bretheren, LISTEN and UNLISTEN). You must
be using PostgreSQL with psycopg2 in your Django project for django-ddp to work. There is no requirement on any
asynchronous framework such as Reddis or crossbar.io as they are simply not needed given the asynchronous support
provided by PostgreSQL with psycopg2. As an added bonus, events dispatched using NOTIFY in a transaction are
only emitted if the transaction is successfully committed.

3
Django DDP Documentation, Release 0.17.0

4 Chapter 1. Requirements
 CHAPTER 2

Scalability

All database queries to support DDP events are done once by the server instance that has made changes via the Django
ORM. Django DDP multiplexes messages for active subscriptions, broadcasting an aggregated change message on
channels specific to each Django model that has been published.
Peer servers subscribe to aggregate broadcast events which are de-multiplexed and dispatched to individual client
connections. No additional database queries are required for de-multiplexing or dispatch by peer servers.

5
Django DDP Documentation, Release 0.17.0

6 Chapter 2. Scalability
 CHAPTER 3

Limitations

No support for the SockJS protocol to support browsers that dont have WebSockets (see
http://caniuse.com/websockets for supported browsers).
Changes must be made via the Django ORM as django-ddp uses Django signals to receive model save/update
signals.

7
Django DDP Documentation, Release 0.17.0

8 Chapter 3. Limitations
 CHAPTER 4

Installation

Install the latest release from pypi (recommended):

pip install django-ddp

Clone and use development version direct from GitHub (to test pre-release code):
pip install -e git+https://github.com/commoncode/django-ddp@develop#egg=django-ddp

9
Django DDP Documentation, Release 0.17.0

10 Chapter 4. Installation
 CHAPTER 5

Example usage

Add dddp to your settings.INSTALLED_APPS:

# settings.py
...
INSTALLED_APPS = list(INSTALLED_APPS) + ['dddp']

If youd like support for the Meteor Accounts package (ie: login/logout with django.contrib.auth) consult the section
on authentication below and use the following line instead:
# settings.py
...
INSTALLED_APPS = list(INSTALLED_APPS) + ['dddp', 'dddp.accounts']

Add ddp.py to your Django application:

# bookstore/ddp.py

from dddp.api import API, Collection, Publication

from bookstore import models

class Book(Collection):
model = models.Book

class Author(Collection):
model = models.Author

class AllBooks(Publication):
queries = [
models.Author.objects.all(),
models.Book.objects.all(),
]

class BooksByAuthorEmail(Publication):
def get_queries(self, author_email):
return [
models.Author.objects.filter(
email=author_email,
),
models.Book.objects.filter(
author__email=author_email,

11
Django DDP Documentation, Release 0.17.0

),
]

API.register(
[Book, Author, AllBooks, BooksByAuthorEmail]
)

Start the Django DDP service:

DJANGO_SETTINGS_MODULE=myproject.settings dddp

12 Chapter 5. Example usage

 CHAPTER 6

Using django-ddp as a secondary DDP connection (RAPID

DEVELOPMENT)

Running in this manner allows rapid development through use of the hot code push features provided by Meteor.
Connect your Meteor application to the Django DDP service:
// bookstore.js
if (Meteor.isClient) {
// Connect to Django DDP service
Django = DDP.connect('http://'+window.location.hostname+':8000/');
// Create local collections for Django models received via DDP
Authors = new Mongo.Collection("bookstore.author", {connection: Django});
Books = new Mongo.Collection("bookstore.book", {connection: Django});
// Subscribe to all books by Janet Evanovich
Django.subscribe('BooksByAuthorEmail', 'janet@evanovich.com');
}

Start Meteor (from within your meteor application directory):

meteor

13
Django DDP Documentation, Release 0.17.0

14 Chapter 6. Using django-ddp as a secondary DDP connection (RAPID DEVELOPMENT)

 CHAPTER 7

Using django-ddp as the primary DDP connection (RECOMMENDED)

If youd prefer to not have two DDP connections (one to Meteor and one to django-ddp) you can set the
DDP_DEFAULT_CONNECTION_URL environment variable to use the specified URL as the primary DDP connec-
tion in Meteor. When doing this, you wont need to use DDP.connect(...) or specify {connection: Django} on your
collections. Running with django-ddp as the primary connection is recommended, and indeed required if you wish to
use dddp.accounts to provide authentication using django.contrib.auth to your meteor app.
DDP_DEFAULT_CONNECTION_URL=http://localhost:8000/ meteor

15
Django DDP Documentation, Release 0.17.0

16 Chapter 7. Using django-ddp as the primary DDP connection (RECOMMENDED)

 CHAPTER 8

Serving your Meteor applications from django-ddp

First, you will need to build your meteor app into a directory (examples below assume target directory named myapp):
meteor build ../myapp

Then, add a MeteorView to your urls.py:

from dddp.views import MeteorView

urlpatterns = patterns(
url('^(?P<path>/.*)$', MeteorView.as_view(
json_path=os.path.join(
settings.PROJ_ROOT, 'myapp', 'bundle', 'star.json',
),
),
)

17
Django DDP Documentation, Release 0.17.0

18 Chapter 8. Serving your Meteor applications from django-ddp

 CHAPTER 9

Adding API endpoints (server method definitions)

API endpoints can be added by calling register method of the dddp.api.API object from the ddp.py module of your
Django app, on a subclass of dddp.api.APIMixin - both dddp.api.Collection and dddp.api.Publication are suitable, or
you may define your own subclass of dddp.api.APIMixin. A good example of this can be seen in dddp/accounts/ddp.py
in the source of django-ddp.

19
Django DDP Documentation, Release 0.17.0

20 Chapter 9. Adding API endpoints (server method definitions)

 CHAPTER 10

Authentication

Authentication is provided using the standard meteor accounts system, along with the accounts-secure package which
turns off Meteors password hashing in favour of using TLS (HTTPS + WebSockets). This ensures strong protection
for all data over the wire. Correctly using TLS/SSL also protects your site against man-in-the-middle and replay
attacks - Meteor is vulnerable to both of these without using encryption.
Add dddp.accounts to your settings.INSTALLED_APPS as described in the example usage section above, then add
tysonclugg:accounts-secure to your Meteor application (from within your meteor application directory):
meteor add tysonclugg:accounts-secure

Then follow the normal procedure to add login/logout views to your Meteor application.

21
Django DDP Documentation, Release 0.17.0

22 Chapter 10. Authentication

 CHAPTER 11

Contributors

Tyson Clugg
Author, conceptual design.
Yan Le
Validate and bug fix dddp.accounts submodule.
MEERQAT
Project sponsor - many thanks for allowing this to be released under an open source license!
David Burles
Expert guidance on how DDP works in Meteor.
Brenton Cleeland
Great conversations around how collections and publications can limit visibility of published documents
to specific users.
Muhammed Thanish
Making the DDP Test Suite available.
This project is forever grateful for the love, support and respect given by the awesome team at Common Code.

23
Django DDP Documentation, Release 0.17.0

24 Chapter 11. Contributors

 CHAPTER 12

Installation:

Django DDP is available for installation direct from PyPi:

pip install django-ddp

25
Django DDP Documentation, Release 0.17.0

26 Chapter 12. Installation:

 CHAPTER 13

Links

The latest documentation is available online at https://django-ddp.readthedocs.org/.

Source code is available online at https://github.com/commoncode/django-ddp.

27
Django DDP Documentation, Release 0.17.0

28 Chapter 13. Links

 CHAPTER 14

Contents

14.1 Change Log

14.1.1 0.17.0

Make the SQL for subscriptions much faster for PostgreSQL.

Repeatable builds using ye olde make.
Use tox test runner - no tests yet (#11).
Add concrete requirement files for test suite (#11).
Started documentation using Sphinx (#10).
Python 3 style exception handling.

14.1.2 0.16.0

New setting: DDP_API_ENDPOINT_DECORATORS. This setting takes a list of dotted import paths to dec-
orators which are applied to API endpoints. For example, enable New Relic instrumentation by adding the
line below to your Django settings.py:
DDP_API_ENDPOINT_DECORATORS = ['newrelic.agent.background_task']

Fixed #7 Warn if using DB engines other than psycopg2 - thanks @Matvey-Kuk.

Improvements to error/exception handling.
Warn if many TX chunks are queued in case WebSocket has stalled.
Bugfix thread locals setup when opening WebSocket.
Add missing import for print function (Python 2).
Work towards #16 Use psycopg2cffi compatibility if psycopg2 not installed.

14.1.3 0.15.0

Renamed Logs collection and publication to dddp.logs to be consistent with naming conventions used elsewhere.
Pass all attributes from logging.LogRecord via dddp.logs collection.

29
Django DDP Documentation, Release 0.17.0

Use select_related() and resultant cached relational fields to speed up Colleciton.serialize() by significantly
reducing round-trips to the database.
Fix bug in get_meteor_ids() which caused many extra database hits.

14.1.4 0.14.0

Correctly handle serving app content from the root path of a domain.
Account security tokens are now calculated for each minute allowing for finer grained token expiry.
Fix bug in error handling where invalid arguments were being passed to logging.error().
Change setting names (and implied meanings):
DDP_PASSWORD_RESET_DAYS_VALID becomes DDP_PASSWORD_RESET_MINUTES_VALID.
DDP_LOGIN_RESUME_DAYS_VALID becomes DDP_LOGIN_RESUME_MINUTES_VALID.
Include created field in logs collection.
Stop depending on Referrer HTTP header which is optional.
Honour verbosity in dddp command, now showing API endpoints in more verbose modes.
Updated dddp.test to Meteor 1.2 and also showing example of URL config to serve Meteor files from Python.

14.1.5 0.13.0

Abstract DDPLauncher out from dddp.main.serve to permit use from other contexts.
Allow Ctrl-C (Break) handling at any time.
Only run async DB connection when PostgresGreenlet is running.
Remove unused import os.path from setup.
Include name and levelno attributes in DDP emitted log records.
Dont attempt to monkey patch more than once.
Include exception info in logger.error logging call.
Update project classifiers to show specific versions of supported dependencies (fixes #6).
Use sane default options for python setup.py bdist_wheel.
Fixed README link to meteor - thanks @LegoStormtroopr.

14.1.6 0.12.2

Set blank=True on AleaIdField, allowing adding items without inventing IDs yourself.

14.1.7 0.12.1

Add AleaIdMixin which provides aid = AleaIdField(unique=True) to models.

Use AleaIdField(unique=True) wherever possible when translating between Meteor style identifiers and Django
primary keys, reducing round trips to the database and hence drastically improving performance when such
fields are available.

30 Chapter 14. Contents

 Django DDP Documentation, Release 0.17.0

14.1.8 0.12.0

Get path to star.json from view config (defined in your urls.py) instead of from settings.
Dropped dddp.server.views, use dddp.views instead.

14.1.9 0.11.0

Support more than 8KB of change data by splitting large payloads into multiple chunks.

14.1.10 0.10.2

Add Logs publication that can be configured to emit logs via DDP through the use of the
dddp.logging.DDPHandler log handler.
Add option to dddp daemon to provide a BackdoorServer (telnet) for interactive debugging (REPL) at runtime.

14.1.11 0.10.1

Bugfix dddp.accounts forgot_password feature.

14.1.12 0.10.0

Stop processing request middleware upon connection - see https://github.com/commoncode/django-

ddp/commit/e7b38b89db5c4e252ac37566f626b5e9e1651a29 for rationale. Access to this.request.user is gone.
Add this.user handling to dddp.accounts.

14.1.13 0.9.14

Fix ordering of user added vs login ready in dddp.accounts authentication methods.

14.1.14 0.9.13

Add dddp.models.get_object_ids helper function.

Add ObjectMappingMixini abstract model mixin providing GenericRelation back to ObjectMapping model.

14.1.15 0.9.12

Bugfix /app.model/schema helper method on collections to work with more model field types.

14.1.16 0.9.11

Fix bug in post login/logout subscription handling.

14.1. Change Log 31

Django DDP Documentation, Release 0.17.0

14.1.17 0.9.10

Fix bug in Accounts.forgotPassword implementation.

14.1.18 0.9.9

Match return values for Accounts.changePassword and Accounts.changePassword methods in dddp.accounts

submodule.

14.1.19 0.9.8

Fix method signature for Accouts.changePassword.

14.1.20 0.9.7

Updated Accounts hashing to prevent cross-purposing auth tokens.

14.1.21 0.9.6

Correct method signature to match Meteor Accounts.resetPassword in dddp.accounts submodule.

14.1.22 0.9.5

Include array of permissions on User publication.

14.1.23 0.9.4

Use mimetypes module to correctly guess mime types for Meteor files being served.

14.1.24 0.9.3

Include ROOT_URL_PATH_PREFIX in ROOT_URL when serving Meteor build files.

14.1.25 0.9.2

Use HTTPS for DDP URL if settings.SECURE_SSL_REDIRECT is set.

14.1.26 0.9.1

Added support for django.contrib.postres.fields.ArrayField serialization.

32 Chapter 14. Contents

 Django DDP Documentation, Release 0.17.0

14.1.27 0.9.0

Added Django 1.8 compatibility. The current implementation has a hackish (but functional) implementation to
use PostgreSQLs array_agg function. Pull requests are welcome.
Retained compatibility with Django 1.7, though we still depend on the dbarray package for this even though not
strictly required with Django 1.8. Once again, pull requests are welcome.

14.1.28 0.8.1

Add missing dependency on pybars3 used to render boilerplate HTML template when serving Meteor applica-
tion files.

14.1.29 0.8.0

Add dddp.server Django app to serve Meteor application files.

Show input params after traceback if exception occurs in API methods.
Small pylint cleanups.

14.1.30 0.7.0

Refactor serialization to improve performance through reduced number of database queries, especially on
sub/unsub.
Fix login/logout user subscription, now emitting user added/ removed upon login/logout respectively.

14.1.31 0.6.5

Use OrderedDict for geventwebsocket.Resource spec to support geventwebsockets 0.9.4 and above.

14.1.32 0.6.4

Send removed messages when client unsubscribes from publications.

Add support for SSL options and settings=SETTINGS args in dddp tool.
Add optional and label attributes to ManyToManyField simple schema.
Check order of added/changed when emitting WebSocket frames rather than when queuing messages.
Move test projects into path that can be imported post install.

14.1.33 0.6.3

Refactor pub/sub functionality to fix support for removed messages.

14.1.34 0.6.2

Bugfix issue where DDP connection thread stops sending messages after changing item that has subscribers for
other connections but not self.

14.1. Change Log 33

Django DDP Documentation, Release 0.17.0

14.1.35 0.6.1

Fix createUser method to login new user after creation.

Dump stack trace to console on error for easier debugging DDP apps.
Fix handing of F expressions in object change handler.
Send nosub in response to invalid subscription request.
Per connection tracking of sent objects so changed/added sent appropriately.

14.1.36 0.6.0

Add dddp.accounts module which provides password based auth mapping to django.contrib.auth module.
Fix ordering of change messages and result message in method calls.

14.1.37 0.5.0

Drop relations to sessions.Session as WebSocket requests dont have HTTP cookie support you must mi-
grate your database after upgrading.
Refactor core to support custom serialization per collection, and correctly dispatch change messages per collec-
tion.
Allow specifying specific collection for publication queries rather than assuming the auto-named default collec-
tions.
Improve schema introspection to include options for fields with choices.
Cleanup transaction handling to apply once at the entry point for DDP API calls.

14.1.38 0.4.0

Make live updates honour user_rel restrictions, also allow superusers to see everything.
Support serializing objects that are saved with F expressions by reading field values for F expressions from
database explicitly before serializing.
Allow fresh connections from browsers that have not established a session in the database yet, also allow sub-
scriptions from unauthenticated sessions (but dont show any data for collections that have user_rel items de-
fined). This change includes a schema change, remember to run migrations after updating.

14.1.39 0.3.0

New DB field: Connection.server_addr you must migrate your database after upgrading.
Cleanup connections on shutdown (and purge associated subscriptions).
Make dddp management command a subclass of the runserver command so that staticfiles work as expected.
Fix non-threadsafe failure in serializer - now using thread local serializer instance.
Fix unsubscribe from publications.
Fix /schema method call.

34 Chapter 14. Contents

 Django DDP Documentation, Release 0.17.0

14.1.40 0.2.5

Fix foreign key references in change messages to correctly reference related object rather than source object.

14.1.41 0.2.4

Fix unicode rendering bug in DDP admin for ObjectMapping model.

14.1.42 0.2.3

Add dddp console script to start DDP service in more robust manner than using the dddp Django mangement
command.

14.1. Change Log 35

Django DDP Documentation, Release 0.17.0

36 Chapter 14. Contents

 CHAPTER 15

Indices and tables

genindex
modindex
search

37
Django DDP Documentation, Release 0.17.0

38 Chapter 15. Indices and tables

 CHAPTER 16

License

The MIT License (MIT)

Copyright (c) 2015 Tyson Clugg
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documen-
tation files (the Software), to deal in the Software without restriction, including without limitation the rights to use,
copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom
the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the
Software.
THE SOFTWARE IS PROVIDED AS IS, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED,
INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PAR-
TICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFT-
WARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

39