How To Export All Keys And Values From Memcached With Python-memcache?

I would like to export all keys and values from a memcached server, using python-memcache. There is no such function in that module. How to do it then? Perhaps something more compl

Solution 1:

This will get you all the keys on a memcached server, you can use any memcached client library to get the value for each key.

import telnetlib

defget_all_memcached_keys(host='127.0.0.1', port=11211):
    t = telnetlib.Telnet(host, port)
    t.write('stats items STAT items:0:number 0 END\n')
    items = t.read_until('END').split('\r\n')
    keys = set()
    for item in items:
        parts = item.split(':')
        ifnotlen(parts) >= 3:
            continue
        slab = parts[1]
        t.write('stats cachedump {} 200000 ITEM views.decorators.cache.cache_header..cc7d9 [6 b; 1256056128 s] END\n'.format(slab))
        cachelines = t.read_until('END').split('\r\n')
        for line in cachelines:
            parts = line.split(' ')
            ifnotlen(parts) >= 3:
                continue
            keys.add(parts[1])
    t.close()
    return keys

Solution 2:

use the memdump and memcat utilities from the libmemcached suite. They can't guarantee you'll get all the data but they're easy to use.

Note: On ubuntu/debian you can get these by intstalling the libmemcached-tools package and they are called memcdump and memccat.

dump all keys:

memcdump --servers=localhost

dump all values:

memccat --servers=localhost `memcdump --servers=localhost`

of course you still have to match up the keys and values - I'd suggest dumping the keys to a file and then using that as the input for memcat (this ensures consistency). Then of course you need to split the values - a full stop is the delimiter I believe - and then pair the keys and values sequentially. There's probably a script out there somewhere...

Solution 3:

There is no way to do that. Memcache protocol does not define any command to iterate over keys. You have to know the key to retrieve value.

Solution 4:

The easiest way is to use python-memcached-stats package, https://github.com/abstatic/python-memcached-stats

The keys() method should get you going.

Solution 5:

As mentioned by others in many places, in the general case, there is no way to list all the keys that stored in a memcached instance. E.g., Memcached: List all keys, Couldn't retrieve all the memcache keys via telnet client

You can, however, list something like the first 1Meg of keys, which is usually enough to have an idea about what are stored in memcache server during development. Basically, you can have two option to extract items from memcache server:

(1) To retrieve a subset of keys and values, you can the method introduced above by use @lrd

However, when the data is very large (e.g., millions of records), this method can be very time-consuming. What's more, this method can only retrieve only a subset of the keys & values.

(2) In cases where you would like to iterate all the items of the memcached server, logging the keys when one adds/set/cas items to memcache server is a much cheaper solution. Then you can read through the log file to get all the keys and get the values from memcache server. As discussed in this mailing list: List all objects in memcached