Zict: Composable Mutable Mappings

The dictionary / mutable mapping interface is powerful and multi-faceted.

  • We store data in different locations such as in-memory, on disk, in archive files, etc..
  • We manage old data with different policies like LRU, random eviction, etc..
  • We might encode or transform data as it arrives or departs the dictionary through compression, encoding, etc..

To this end we build abstract MutableMapping classes that consume and build on other MutableMappings. We can compose several of these with each other to form intuitive interfaces over complex storage systems policies.

Example

In the following example we create an LRU dictionary backed by pickle-encoded, zlib-compressed, directory of files.

import pickle
import zlib

from zict import File, Func, LRU

a = File('myfile/', mode='a')
b = Func(zlib.compress, zlib.decompress, a)
c = Func(pickle.dumps, pickle.loads, b)
d = LRU(100, c)

>>> d['x'] = [1, 2, 3]
>>> d['x']
[1, 2, 3]

API

class zict.lru.LRU(n, d, on_evict=<function do_nothing>, weight=<function <lambda>>)[source]

Evict Least Recently Used Elements

Parameters:

n: int

Number of elements to keep, or total weight if weight= is used

d: MutableMapping

Dictionary in which to hold elements

on_evict: callable

Function:: k, v -> action to call on key value pairs prior to eviction

weight: callable

Function:: k, v -> number to determine the size of keeping the item in the mapping. Defaults to (k, v) -> 1

Examples

>>> lru = LRU(2, dict(), on_evict=lambda k, v: print("Lost", k, v))
>>> lru['x'] = 1
>>> lru['y'] = 2
>>> lru['z'] = 3
Lost x 1

Methods

class zict.file.File(directory, mode='a')[source]

Mutable Mapping interface to a directory

Keys must be strings, values must be bytes

Parameters:

directory: string

mode: string, (‘r’, ‘w’, ‘a’), defaults to ‘a’

Examples

>>> z = File('myfile')  
>>> z['x'] = b'123'  
>>> z['x']  
b'123'

Methods

class zict.func.Func(dump, load, d)[source]

Buffer a MutableMapping with a pair of input/output functions

Parameters:

dump: callable

Function to call on value as we set it into the mapping

load: callable

Function to call on value as we pull it from the mapping

d: MutableMapping

Examples

>>> def double(x):
...     return x * 2
>>> def halve(x):
...     return x / 2
>>> d = dict()
>>> f = Func(double, halve, d)
>>> f['x'] = 10
>>> d
{'x': 20}
>>> f['x']
10.0

Methods

class zict.zip.Zip(filename, mode='a')[source]

Mutable Mapping interface to a Zip file

Keys must be strings, values must be bytes

Parameters:

filename: string

mode: string, (‘r’, ‘w’, ‘a’), defaults to ‘a’

Examples

>>> z = Zip('myfile.zip')  
>>> z['x'] = b'123'  
>>> z['x']  
b'123'
>>> z.flush()  # flush and write metadata to disk  

Attributes

Methods