Collection Configuration and Techniques

The default behavior of relationship() is to fully load the collection of items in, as according to the loading strategy of the relationship. Additionally, the by default only knows how to delete objects which are actually present within the session. When a parent instance is marked for deletion and flushed, the Session loads its full list of child items in so that they may either be deleted as well, or have their foreign key value set to null; this is to avoid constraint violations. For large collections of child items, there are several strategies to bypass full loading of child items both at load time as well as deletion time.

Note

This is a legacy feature. Using the filter in conjunction with select() is the method of use. For relationships that shouldn’t load, set relationship.lazy to noload.

A which corresponds to a large collection can be configured so that it returns a legacy Query object when accessed, which allows filtering of the relationship on criteria. The class is a special class returned in place of a collection when accessed. Filtering criterion may be applied as well as limits and offsets, either explicitly or via array slices:

The dynamic relationship supports limited write operations, via the AppenderQuery.append() and AppenderQuery.remove() methods:

oldpost = jack.posts.filter(Post.headline=='old post').one()
jack.posts.remove(oldpost)
jack.posts.append(Post('new post'))

Since the read side of the dynamic relationship always queries the database, changes to the underlying collection will not be visible until the data has been flushed. However, as long as “autoflush” is enabled on the Session in use, this will occur automatically each time the collection is about to emit a query.

To place a dynamic relationship on a backref, use the function in conjunction with lazy='dynamic':

class Post(Base):
    __table__ = posts_table
    user = relationship(User,
                backref=backref('posts', lazy='dynamic')
            )

Note that eager/lazy loading options cannot be used in conjunction dynamic relationships at this time.

class sqlalchemy.orm.``AppenderQuery(attr, state)

A dynamic query that supports basic collection storage operations.

Class signature

class sqlalchemy.orm.AppenderQuery (sqlalchemy.orm.dynamic.AppenderMixin, )

Note

The dynamic_loader() function is essentially the same as with the lazy='dynamic' argument specified.

Warning

The “dynamic” loader applies to collections only. It is not valid to use “dynamic” loaders with many-to-one, one-to-one, or uselist=False relationships. Newer versions of SQLAlchemy emit warnings or exceptions in these cases.

Setting Noload, RaiseLoad

A “noload” relationship never loads from the database, even when accessed. It is configured using lazy='noload':

class MyClass(Base):
    __tablename__ = 'some_table'
    children = relationship(MyOtherClass, lazy='noload')

Above, the children collection is fully writeable, and changes to it will be persisted to the database as well as locally available for reading at the time they are added. However when instances of MyClass are freshly loaded from the database, the children collection stays empty. The noload strategy is also available on a query option basis using the loader option.

Alternatively, a “raise”-loaded relationship will raise an InvalidRequestError where the attribute would normally emit a lazy load:

class MyClass(Base):
    __tablename__ = 'some_table'
    children = relationship(MyOtherClass, lazy='raise')

Above, attribute access on the children collection will raise an exception if it was not previously eagerloaded. This includes read access but for collections will also affect write access, as collections can’t be mutated without first loading them. The rationale for this is to ensure that an application is not emitting any unexpected lazy loads within a certain context. Rather than having to read through SQL logs to determine that all necessary attributes were eager loaded, the “raise” strategy will cause unloaded attributes to raise immediately if accessed. The raise strategy is also available on a query option basis using the loader option.

New in version 1.1: added the “raise” loader strategy.

Dictionary Collections

A little extra detail is needed when using a dictionary as a collection. This because objects are always loaded from the database as lists, and a key-generation strategy must be available to populate the dictionary correctly. The function is by far the most common way to achieve a simple dictionary collection. It produces a dictionary class that will apply a particular attribute of the mapped class as a key. Below we map an Item class containing a dictionary of Note items keyed to the Note.keyword attribute:

from sqlalchemy import Column, Integer, String, ForeignKey
from sqlalchemy.orm import relationship
from sqlalchemy.orm.collections import attribute_mapped_collection
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
class Item(Base):
    __tablename__ = 'item'
    id = Column(Integer, primary_key=True)
    notes = relationship("Note",
                collection_class=attribute_mapped_collection('keyword'),
                cascade="all, delete-orphan")
class Note(Base):
    __tablename__ = 'note'
    id = Column(Integer, primary_key=True)
    item_id = Column(Integer, ForeignKey('item.id'), nullable=False)
    keyword = Column(String)
    text = Column(String)
    def __init__(self, keyword, text):
        self.keyword = keyword
        self.text = text

Item.notes is then a dictionary:

>>> item = Item()
>>> item.notes['a'] = Note('a', 'atext')
>>> item.notes.items()
{'a': <__main__.Note object at 0x2eaaf0>}

attribute_mapped_collection() will ensure that the .keyword attribute of each Note complies with the key in the dictionary. Such as, when assigning to Item.notes, the dictionary key we supply must match that of the actual Note object:

item = Item()
item.notes = {
            'a': Note('a', 'atext'),
            'b': Note('b', 'btext')
        }

The attribute which uses as a key does not need to be mapped at all! Using a regular Python @property allows virtually any detail or combination of details about the object to be used as the key, as below when we establish it as a tuple of Note.keyword and the first ten letters of the Note.text field:

class Item(Base):
    __tablename__ = 'item'
    id = Column(Integer, primary_key=True)
    notes = relationship("Note",
                collection_class=attribute_mapped_collection('note_key'),
                backref="item",
                cascade="all, delete-orphan")
class Note(Base):
    __tablename__ = 'note'
    id = Column(Integer, primary_key=True)
    item_id = Column(Integer, ForeignKey('item.id'), nullable=False)
    keyword = Column(String)
    text = Column(String)
    @property
    def note_key(self):
        return (self.keyword, self.text[0:10])
    def __init__(self, keyword, text):
        self.keyword = keyword
        self.text = text

Above we added a Note.item backref. Assigning to this reverse relationship, the Note is added to the Item.notes dictionary and the key is generated for us automatically:

>>> n1 = Note("a", "atext")
>>> n1.item = item
>>> item.notes
{('a', 'atext'): <__main__.Note object at 0x2eaaf0>}

Other built-in dictionary types include column_mapped_collection(), which is almost like except given the Column object directly:

from sqlalchemy.orm.collections import column_mapped_collection
class Item(Base):
    __tablename__ = 'item'
    id = Column(Integer, primary_key=True)
    notes = relationship("Note",
                collection_class=column_mapped_collection(Note.__table__.c.keyword),
                cascade="all, delete-orphan")

as well as which is passed any callable function. Note that it’s usually easier to use along with a @property as mentioned earlier:

from sqlalchemy.orm.collections import mapped_collection
class Item(Base):
    __tablename__ = 'item'
    id = Column(Integer, primary_key=True)
    notes = relationship("Note",
                collection_class=mapped_collection(lambda note: note.text[0:10]),
                cascade="all, delete-orphan")

Dictionary mappings are often combined with the “Association Proxy” extension to produce streamlined dictionary views. See and Composite Association Proxies for examples.

Dealing with Key Mutations and back-populating for Dictionary collections

When using attribute_mapped_collection(), the “key” for the dictionary is taken from an attribute on the target object. Changes to this key are not tracked. This means that the key must be assigned towards when it is first used, and if the key changes, the collection will not be mutated. A typical example where this might be an issue is when relying upon backrefs to populate an attribute mapped collection. Given the following:

Above, if we create a B() that refers to a specific A(), the back populates will then add the B() to the A.bs collection, however if the value of B.data is not set yet, the key will be None:

>>> a1 = A()
>>> b1 = B(a=a1)
>>> a1.bs
{None: <test3.B object at 0x7f7b1023ef70>}

Setting b1.data after the fact does not update the collection:

>>> b1.data = 'the key'
>>> a1.bs
{None: <test3.B object at 0x7f7b1023ef70>}

This can also be seen if one attempts to set up B() in the constructor. The order of arguments changes the result:

>>> B(a=a1, data='the key')
<test3.B object at 0x7f7b10114280>
>>> a1.bs
{None: <test3.B object at 0x7f7b10114280>}

vs:

>>> B(data='the key', a=a1)
<test3.B object at 0x7f7b10114340>
>>> a1.bs
{'the key': <test3.B object at 0x7f7b10114340>}

If backrefs are being used in this way, ensure that attributes are populated in the correct order using an __init__ method.

An event handler such as the following may also be used to track changes in the collection as well:

from sqlalchemy import event
from sqlalchemy.orm import attributes
@event.listens_for(B.data, "set")
def set_item(obj, value, previous, initiator):
    if obj.a is not None:
        previous = None if previous == attributes.NO_VALUE else previous
        obj.a.bs[value] = obj
        obj.a.bs.pop(previous)

function sqlalchemy.orm.collections.``attribute_mapped_collection(attr_name)

A dictionary-based collection type with attribute-based keying.

Returns a factory with a keying based on the ‘attr_name’ attribute of entities in the collection, where attr_name is the string name of the attribute.

Warning

the key value must be assigned to its final value before it is accessed by the attribute mapped collection. Additionally, changes to the key attribute are not tracked automatically, which means the key in the dictionary is not automatically synchronized with the key value on the target object itself. See the section Dealing with Key Mutations and back-populating for Dictionary collections for an example.

function sqlalchemy.orm.collections.``column_mapped_collection(mapping_spec)

A dictionary-based collection type with column-based keying.

Returns a factory with a keying function generated from mapping_spec, which may be a Column or a sequence of Columns.

The key value must be immutable for the lifetime of the object. You can not, for example, map on foreign key values if those key values will change during the session, i.e. from None to a database-assigned integer after a session flush.

function sqlalchemy.orm.collections.``mapped_collection(keyfunc)

A dictionary-based collection type with arbitrary keying.

Returns a MappedCollection factory with a keying function generated from keyfunc, a callable that takes an entity and returns a key value.

You can use your own types for collections as well. In simple cases, inheriting from list or set, adding custom behavior, is all that’s needed. In other cases, special decorators are needed to tell SQLAlchemy more detail about how the collection operates.

Do I need a custom collection implementation?

In most cases not at all! The most common use cases for a “custom” collection is one that validates or marshals incoming values into a new form, such as a string that becomes a class instance, or one which goes a step beyond and represents the data internally in some fashion, presenting a “view” of that data on the outside of a different form.

For the first use case, the decorator is by far the simplest way to intercept incoming values in all cases for the purposes of validation and simple marshaling. See Simple Validators for an example of this.

For the second use case, the extension is a well-tested, widely used system that provides a read/write “view” of a collection in terms of some attribute present on the target object. As the target attribute can be a @property that returns virtually anything, a wide array of “alternative” views of a collection can be constructed with just a few functions. This approach leaves the underlying mapped collection unaffected and avoids the need to carefully tailor collection behavior on a method-by-method basis.

Customized collections are useful when the collection needs to have special behaviors upon access or mutation operations that can’t otherwise be modeled externally to the collection. They can of course be combined with the above two approaches.

The collections package understands the basic interface of lists, sets and dicts and will automatically apply instrumentation to those built-in types and their subclasses. Object-derived types that implement a basic collection interface are detected and instrumented via duck-typing:

class ListLike(object):
    def __init__(self):
        self.data = []
    def append(self, item):
        self.data.append(item)
    def remove(self, item):
        self.data.remove(item)
    def extend(self, items):
        self.data.extend(items)
    def __iter__(self):
        return iter(self.data)
    def foo(self):
        return 'foo'

append, remove, and extend are known list-like methods, and will be instrumented automatically. __iter__ is not a mutator method and won’t be instrumented, and foo won’t be either.

Duck-typing (i.e. guesswork) isn’t rock-solid, of course, so you can be explicit about the interface you are implementing by providing an __emulates__ class attribute:

class SetLike(object):
    __emulates__ = set
    def __init__(self):
        self.data = set()
    def append(self, item):
        self.data.add(item)
    def remove(self, item):
        self.data.remove(item)
    def __iter__(self):
        return iter(self.data)

This class looks list-like because of append, but __emulates__ forces it to set-like. remove is known to be part of the set interface and will be instrumented.

But this class won’t work quite yet: a little glue is needed to adapt it for use by SQLAlchemy. The ORM needs to know which methods to use to append, remove and iterate over members of the collection. When using a type like list or set, the appropriate methods are well-known and used automatically when present. This set-like class does not provide the expected add method, so we must supply an explicit mapping for the ORM via a decorator.

Decorators can be used to tag the individual methods the ORM needs to manage collections. Use them when your class doesn’t quite meet the regular interface for its container type, or when you otherwise would like to use a different method to get the job done.

from sqlalchemy.orm.collections import collection
class SetLike(object):
    __emulates__ = set
    def __init__(self):
        self.data = set()
    @collection.appender
    def append(self, item):
        self.data.add(item)
    def remove(self, item):
        self.data.remove(item)
    def __iter__(self):
        return iter(self.data)

And that’s all that’s needed to complete the example. SQLAlchemy will add instances via the append method. remove and __iter__ are the default methods for sets and will be used for removing and iteration. Default methods can be changed as well:

from sqlalchemy.orm.collections import collection
class MyList(list):
    @collection.remover
    def zark(self, item):
        # do something special...
    @collection.iterator
    def hey_use_this_instead_for_iteration(self):
        # ...

There is no requirement to be list-, or set-like at all. Collection classes can be any shape, so long as they have the append, remove and iterate interface marked for SQLAlchemy’s use. Append and remove methods will be called with a mapped entity as the single argument, and iterator methods are called with no arguments and must return an iterator.

class sqlalchemy.orm.collections.``collection

Decorators for entity collection classes.

The decorators fall into two groups: annotations and interception recipes.

The annotating decorators (appender, remover, iterator, converter, internally_instrumented) indicate the method’s purpose and take no arguments. They are not written with parens:

@collection.appender
def append(self, append): ...

The recipe decorators all require parens, even those that take no arguments:

@collection.adds('entity')
def insert(self, position, entity): ...
@collection.removes_return()
def popitem(self): ...

method sqlalchemy.orm.collections.collection.static adds(arg)

Mark the method as adding an entity to the collection.

Adds “add to collection” handling to the method. The decorator argument indicates which method argument holds the SQLAlchemy-relevant value. Arguments can be specified positionally (i.e. integer) or by name:
```
@collection.adds(1)
@collection.adds('entity')
def do_stuff(self, thing, entity=None): ...
```
method static appender(fn)

Tag the method as the collection appender.

The appender method is called with one positional argument: the value to append. The method will be automatically decorated with ‘adds(1)’ if not already decorated:
```
@collection.appender
def add(self, append): ...
# or, equivalently
@collection.appender
@collection.adds(1)
def add(self, append): ...
# for mapping type, an 'append' may kick out a previous value
# that occupies that slot.  consider d['a'] = 'foo'- any previous
# value in d['a'] is discarded.
@collection.appender
@collection.replaces(1)
def add(self, entity):
    key = some_key_func(entity)
    previous = None
        previous = self[key]
    self[key] = entity
    return previous
```
If the value to append is not allowed in the collection, you may raise an exception. Something to remember is that the appender will be called for each object mapped by a database query. If the database contains rows that violate your collection semantics, you will need to get creative to fix the problem, as access via the collection will not work.

If the appender method is internally instrumented, you must also receive the keyword argument ‘_sa_initiator’ and ensure its promulgation to collection events.
method sqlalchemy.orm.collections.collection.static converter(fn)

Tag the method as the collection converter.

Deprecated since version 1.3: The handler is deprecated and will be removed in a future release. Please refer to the bulk_replace listener interface in conjunction with the listen() function.

This optional method will be called when a collection is being replaced entirely, as in:

The converter method will receive the object being assigned and should return an iterable of values suitable for use by the appender method. A converter must not assign values or mutate the collection, its sole job is to adapt the value the user provides into an iterable of values for the ORM’s use.

The default converter implementation will use duck-typing to do the conversion. A dict-like collection will be convert into an iterable of dictionary values, and other types will simply be iterated:
```
@collection.converter
def convert(self, other): ...
```
If the duck-typing of the object does not match the type of this collection, a TypeError is raised.

Supply an implementation of this method if you want to expand the range of possible types that can be assigned in bulk or perform validation on the values about to be assigned.
method static internally_instrumented(fn)

Tag the method as instrumented.

This tag will prevent any decoration from being applied to the method. Use this if you are orchestrating your own calls to collection_adapter() in one of the basic SQLAlchemy interface methods, or to prevent an automatic ABC method decoration from wrapping your implementation:
```
# normally an 'extend' method on a list-like class would be
# automatically intercepted and re-implemented in terms of
# SQLAlchemy events and append().  your implementation will
# never be called, unless:
@collection.internally_instrumented
def extend(self, items): ...
```
method sqlalchemy.orm.collections.collection.static iterator(fn)

Tag the method as the collection remover.

The iterator method is called with no arguments. It is expected to return an iterator over all collection members:
```
@collection.iterator
def __iter__(self): ...
```
method static remover(fn)

Tag the method as the collection remover.

The remover method is called with one positional argument: the value to remove. The method will be automatically decorated with removes_return() if not already decorated:
```
@collection.remover
def zap(self, entity): ...
# or, equivalently
@collection.remover
@collection.removes_return()
def zap(self, ): ...
```
If the value to remove is not present in the collection, you may raise an exception or return None to ignore the error.

If the remove method is internally instrumented, you must also receive the keyword argument ‘_sa_initiator’ and ensure its promulgation to collection events.
method static removes(arg)

Mark the method as removing an entity in the collection.

Adds “remove from collection” handling to the method. The decorator argument indicates which method argument holds the SQLAlchemy-relevant value to be removed. Arguments can be specified positionally (i.e. integer) or by name:
```
@collection.removes(1)
def zap(self, item): ...
```
For methods where the value to remove is not known at call-time, use collection.removes_return.
method sqlalchemy.orm.collections.collection.static removes_return()

Mark the method as removing an entity in the collection.

Adds “remove from collection” handling to the method. The return value of the method, if any, is considered the value to remove. The method arguments are not inspected:
```
@collection.removes_return()
def pop(self): ...
```
For methods where the value to remove is known at call-time, use collection.remove.
method static replaces(arg)

Mark the method as replacing an entity in the collection.

Adds “add to collection” and “remove from collection” handling to the method. The decorator argument indicates which method argument holds the SQLAlchemy-relevant value to be added, and return value, if any will be considered the value to remove.

Arguments can be specified positionally (i.e. integer) or by name:
```
@collection.replaces(2)
def __setitem__(self, index, item): ...
```

Custom Dictionary-Based Collections

The class can be used as a base class for your custom types or as a mix-in to quickly add dict collection support to other classes. It uses a keying function to delegate to __setitem__ and __delitem__:

from sqlalchemy.util import OrderedDict
from sqlalchemy.orm.collections import MappedCollection
class NodeMap(OrderedDict, MappedCollection):
    """Holds 'Node' objects, keyed by the 'name' attribute with insert order maintained."""
    def __init__(self, *args, **kw):
        MappedCollection.__init__(self, keyfunc=lambda node: node.name)
        OrderedDict.__init__(self, *args, **kw)

When subclassing MappedCollection, user-defined versions of __setitem__() or __delitem__() should be decorated with , if they call down to those same methods on MappedCollection. This because the methods on are already instrumented - calling them from within an already instrumented call can cause events to be fired off repeatedly, or inappropriately, leading to internal state corruption in rare cases:

from sqlalchemy.orm.collections import MappedCollection,\
                                    collection
class MyMappedCollection(MappedCollection):
    """Use @internally_instrumented when your methods
    call down to already-instrumented methods.
    """
    @collection.internally_instrumented
    def __setitem__(self, key, value, _sa_initiator=None):
        # do something with key, value
        super(MyMappedCollection, self).__setitem__(key, value, _sa_initiator)
    @collection.internally_instrumented
    def __delitem__(self, key, _sa_initiator=None):
        # do something with key
        super(MyMappedCollection, self).__delitem__(key, _sa_initiator)

The ORM understands the dict interface just like lists and sets, and will automatically instrument all dict-like methods if you choose to subclass dict or provide dict-like collection behavior in a duck-typed class. You must decorate appender and remover methods, however- there are no compatible methods in the basic dictionary interface for SQLAlchemy to use by default. Iteration will go through itervalues() unless otherwise decorated.

Note

Due to a bug in MappedCollection prior to version 0.7.6, this workaround usually needs to be called before a custom subclass of MappedCollection which uses can be used:

from sqlalchemy.orm.collections import _instrument_class, MappedCollection
_instrument_class(MappedCollection)

This will ensure that the MappedCollection has been properly initialized with custom __setitem__() and __delitem__() methods before used in a custom subclass.

class sqlalchemy.orm.collections.``MappedCollection(keyfunc)

A basic dictionary-based collection class.

Extends dict with the minimal bag semantics that collection classes require. set and remove are implemented in terms of a keying function: any callable that takes an object and returns an object for use as a dictionary key.

Class signature

class (builtins.dict)

method sqlalchemy.orm.collections.MappedCollection.__init__(keyfunc)

keyfunc may be any callable that takes an object and returns an object for use as a dictionary key.

The keyfunc will be called every time the ORM needs to add a member by value-only (such as when loading instances from the database) or remove a member. The usual cautions about dictionary keying apply- keyfunc(object) should return the same output for the life of the collection. Keying based on mutable properties can result in unreachable instances “lost” in the collection.
method clear() → None. Remove all items from D.
method sqlalchemy.orm.collections.MappedCollection.pop(k[, d]) → v, remove specified key and return the corresponding value.

If key is not found, d is returned if given, otherwise KeyError is raised
method popitem() → (k, v), remove and return some (key, value) pair as a

2-tuple; but raise KeyError if D is empty.
method sqlalchemy.orm.collections.MappedCollection.remove(value, _sa_initiator=None)

Remove an item by value, consulting the keyfunc for the key.
method set(value, _sa_initiator=None)

Add an item by value, consulting the keyfunc for the key.
method sqlalchemy.orm.collections.MappedCollection.setdefault(key, default=None)

Insert key with a value of default if key is not in the dictionary.

Return the value for key if key is in the dictionary, else default.
method update([E, ]\*F*) → None. Update D from dict/iterable E and F.

If E is present and has a .keys() method, then does: for k in E: D[k] = E[k] If E is present and lacks a .keys() method, then does: for k, v in E: D[k] = v In either case, this is followed by: for k in F: D[k] = F[k]

Many custom types and existing library classes can be used as a entity collection type as-is without further ado. However, it is important to note that the instrumentation process will modify the type, adding decorators around methods automatically.

The decorations are lightweight and no-op outside of relationships, but they do add unneeded overhead when triggered elsewhere. When using a library class as a collection, it can be good practice to use the “trivial subclass” trick to restrict the decorations to just your usage in relationships. For example:

class MyAwesomeList(some.great.library.AwesomeList):
    pass
# ... relationship(..., collection_class=MyAwesomeList)

The ORM uses this approach for built-ins, quietly substituting a trivial subclass when a list, set or dict is used directly.

Various internal methods.

function sqlalchemy.orm.collections.``bulk_replace(values, existing_adapter, new_adapter, initiator=None)

Load a new collection, firing events based on prior like membership.

Appends instances in values onto the new_adapter. Events will be fired for any instance not present in the existing_adapter. Any instances in existing_adapter not present in values will have remove events fired upon them.

Parameters
- values – An iterable of collection member instances
- existing_adapter – A CollectionAdapter of instances to be replaced
- new_adapter – An empty to load with values

class sqlalchemy.orm.collections.``collection

Decorators for entity collection classes.

The decorators fall into two groups: annotations and interception recipes.

The annotating decorators (appender, remover, iterator, converter, internally_instrumented) indicate the method’s purpose and take no arguments. They are not written with parens:

@collection.appender
def append(self, append): ...

The recipe decorators all require parens, even those that take no arguments:

@collection.adds('entity')
def insert(self, position, entity): ...
@collection.removes_return()

sqlalchemy.orm.collections.``collection_adapter = operator.attrgetter(‘_sa_adapter’)

Fetch the CollectionAdapter for a collection.

class sqlalchemy.orm.collections.``CollectionAdapter(attr, owner_state, data)

Bridges between the ORM and arbitrary Python collections.

Proxies base-level collection operations (append, remove, iterate) to the underlying Python collection, and emits add/remove events for entities entering or leaving the collection.

The ORM uses exclusively for interaction with entity collections.

class sqlalchemy.orm.collections.``InstrumentedDict

An instrumented version of the built-in dict.

Class signature

class sqlalchemy.orm.collections.InstrumentedDict (builtins.dict)

class sqlalchemy.orm.collections.``InstrumentedList(iterable=(), /)

An instrumented version of the built-in list.

Class signature

class (builtins.list)

class sqlalchemy.orm.collections.``InstrumentedSet

An instrumented version of the built-in set.

Class signature