Vibrations of a restless brain

By its very nature every embodied spirit is doomed to suffer and enjoy in solitude. Sensations, feelings, insights, fancies - all these are private and, except through symbols and at second hand, incommunicable. We can pool information about experiences, but never the experiences themselves. From family to nation, every human group is a society of island universe.

Neither agreeable nor disagreeable, it just is.

The suggestion is that the function of the brain and nervous system and sense organs is in the main eliminative and not productive. Each person is at each moment capable of remembering all that has ever happened to him and of perceiving everything that is happening everywhere in the universe. The function of the brain and nervous system is to protect us from being overwhelmed and confused by this mass of largely useless and irrelevant knowledge, by shutting out most of what we should otherwise perceive or remember at any moment, and leaving only that very small and special selection which is likely to be practically useful. 

According to such a theory, each one of us is potentially Mind at Large. But in so far as we are animals, our business is at all costs to survive. To make biological survival possible, Mind at Large has to be funneled through the reducing valve of the brain and nervous system. What comes out at the other end is a measly trickle of the kind of consciousness which will help us to stay alive on the surface of this Particular planet

Most men and women lead lives at the worst so painful, at the best so monotonous, poor and limited that the urge to escape, the longing to transcend themselves if only for a few moments, is and has always been one of the principal appetites of the soul

If you use solr-ruby gem, you should be well aware that Solr attempts to create an XML doc from the provided doc-hash. It first attempts to use ‘xml/libxml’, which if not available, falls back to REXML. It is recommended to use libxml.

All you need to do is

gem install libxml-ruby

If you are lucky enough, that’s all for you. However, many a times we face issues like

Building native extensions.  This could take a while...
ERROR:  Error installing libxml-ruby:
	ERROR: Failed to build gem native extension.

/usr/bin/ruby extconf.rb
checking for socket() in -lsocket... no
checking for gethostbyname() in -lnsl... yes
checking for atan() in -lm... no
checking for atan() in -lm... yes
checking for inflate() in -lz... no
checking for inflate() in -lzlib... no
checking for inflate() in -lzlib1... no
checking for inflate() in -llibz... no
*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of
necessary libraries and/or headers.  Check the mkmf.log file for more
details.  You may need configuration options.

If this is the case, you should install zlib-devel and libxml2-devel packages.

yum -y install zlib-devel
yum -y install libxml2-devel
gem install libxml-ruby

That sorts it all. You are good to go

Recently, One of our servers started to have the following error

irb(main):001:0> require 'rubygems'
=> true
irb(main):002:0> require 'mysql'
=> true
irb(main):003:0> conn = Mysql.connect('db01', 'xxxxxx', 'xxxxxx', 'employee')
=> #
irb(main):004:0> a = conn.query("SELECT * FROM employees WHERE id >= 8500 AND id < 9000")
=> #
irb(main):005:0> a.fetch_hash
ArgumentError: NULL pointer given
    from (irb):5:in `fetch_hash'
	from (irb):5
irb(main):006:0> exit

The method fetch_hash is a standard method and works. Here is the versions of MySQL and ruby that I used

-bash-3.2$ mysql --version
mysql  Ver 14.12 Distrib 5.0.89, for unknown-linux-gnu (x86_64) using readline 5.1

-bash-3.2$ ruby --version
ruby 1.8.7 (2009-06-12 patchlevel 174) [x86_64-linux]

If you face this issue in your servers or anywhere try the following workaround. They have worked for me, and they should work for you as well.

1. uninstall the MySQL-shared-compat package
2. Re Install mysql gem

Controlling activity feeds on facebook

Facebook has a setting which lets you chose amongst your friends and pages, whose updates you want to see and whose you want to skip. Here it is how

Next to Most Recent, there is a drop down button. Click on that to gain access to control options. The drop down menu comes only when you are viewing the Most Recent news feeds only. So if you don’t see an option, just click on Most Recent. The page will refresh itself and the menu options will come.

Select Edit Options in the drop down menu

Now conveniently you can choose to show news from

1. Those friends and pages you interact with most or
2. All friends and pages.

The default setting is “friends and pages” you interact with most. You can also edit friends whose feeds have been blocked in the 

Code Snippet to recursively traverse a dictionary and remove certain key/value pair

In case of complex dictionaries like

my_blog = { '_created' : datetime.datetime (2007,01,03),
          '_updated' : datetime.datetime (2011,06,11),
            'name'     : 'MyBLive',
            'latest_post'    : { '_created' : datetime.datetime (2007,06,9),
                                 '_updated' : datetime.datetime (2007,069),
                                 'name'     : 'Hello World !' }

The problem with this is that datetime.datetime entities are not JSON Serializable. One approach could be to provide a serializer for datetime.datetime entities as suggested by verte over IRC. If you want to have a datetime.datetime aware JSONSerializer, you should have a look at the django.core.serializers.json module

In my case, we were using a Google App Engine application and the JSON response need not contain these key/value pairs so it makes more sense if these are removed from the dictionary.

def rm (d, l):
  """
  Removed from dictionary "d" all those key-value pairs the keys of which
  are defined as a list in "l"
  """
  if not l: return d
  if reduce ( lambda x,y: x or y, [x in d.keys () for x in l]):
    [d.pop (x, None) for x in l]
  [rm (x, l) for x in d.values () if isinstance (x, dict)]
  return d

The usage of memcache for lowering the load on App Engine Datastore is well known. Here is an approach to versioned caching of datastore entities in App Engine. The Model is deigned primarily for purposes where entity is updated very frequently. The idea for these kind of entity generated while working on the Multiuser Chat room for App Engine using Channel APIs.

Expectations

The Model is designed with the following factors in mind

• Very frequent read/write access to entity
• Effective usage of memcache to reduce load on datastore operations
• Strong consistency between datastore and memcache

Design Plan

The Entities have an internal version number. This version number is increased everytime there is an update in the entity.

from google.appengine.ext import db


#   The maximum difference in revisions acceptable  at any instant
#   between memcached values and the datastore values. The higher it
#   is, the greater catastrophe when memcache goes down, but lesser
#   datastore usage. The lesser it is, the more consistent your datastore
#   and memcache are, and higher datastore operations. 4~6
FAULT_TOLERANCE = 4

class GlobalVersionedCachingModel(db.Model):
    """
    The Model uses internal versioning of information with prime focus on very
    high read/writes and consistency.
    Every entity has a datastore's version number information and the version
    number from memcache. When the entity is updated, it happens in memcache
    only and the memcached version number increases. If this number is greater
    than the datastore version number by a certain amount called
    "fault tolerance", then the datastor entity is sync'd with the memcache
    entity.
    """

    _db_version = db.IntegerProperty (default=0, required=True)
    _cache_version = db.IntegerProperty (default=0, required=True)
    _fault_tolerance = db.IntegerProperty(default = FAULT_TOLERANCE)

Approach #1

Initially, both the versions are set to 0 when the entity is created. _fault_tolerance is the maximum allowed difference between the cache version and the datastore version. Since the entity is primarily read from and written to memcache, and later updated to datastore, the datastore version number can be behind the memcache version number. When the difference between the two exceeds fault tolerance, then the datastore is updated with the memcache details.

The downside of this approach is that when the memcache goes out, then the datastore entity is fetched. This entity, in worst case scenario, could be lagging from the actual entity by a maximum of _fault_tolerance factor. Fault Tolerance can be decreased to improve the consistency between memcached entity and the datastore entity but that will result in higher datastore read/write operations.

Approach #2

In this approach, instead of directly writing into the datastore, we initiate a task queue and that gets the task done for us. The good part of this approach is that we can keep smaller values of _fault_tolerance and still expect faster processing. This approach is ideal where strong consistency is required between the entities and latency shall also be minimal

Methods

def get2 (keys, **kwargs):
    keys, multiple = datastore.NormalizeAndTypeCheckKeys (keys)
    getted_cache = memcache.get_multi (map (str, keys))
    ret = map (deserialize_entities, getted_cache.values ())
    keys_to_fetch = [key for key in keys if getted_cache.get(key, None) is not None]
    getted_db = db.get(keys_to_fetch)
    memcache_to_set = dict ((k,v) for k,v in zip (map (str,keys_to_fetch), 
                                            map (serialize_entities, getted_db)))
    ret.extend(getted_db)
    memcache.set_multi (memcache_to_set)
    if multiple:
        return ret
    if len (ret) > 0:
        return ret[0]

class GlobalVersionedCachingModel(db.Model):
    """
    The Model uses internal versioning of information with prime focus on very
    high read/writes and consistency.
    Every entity has a datastore's version number information and the version
    number from memcache. When the entity is updated, it happens in memcache
    only and the memcached version number increases. If this number is greater
    than the datastore version number by a certain amount called
    "fault tolerance", then the datastor entity is sync'd with the memcache
    entity.
    """

    _db_version = db.IntegerProperty (default=0, required=True)
    _cache_version = db.IntegerProperty (default=0, required=True)
    _fault_tolerance = db.IntegerProperty(default = FAULT_TOLERANCE)
    created = db.DateTimeProperty (auto_now_add=True)
    updated = db.DateTimeProperty (auto_now=True)

    @property
    def keyname (self):
        return str (self.key ())

    def remove_from_cache (self, update_db=False):
        """
        Removes the cached instance of the entity. If update_db is True,
        then updates the datastore before removing from cache so that no data
        is lost.
        """
        if update_db:
            self.update_to_db()
        memcache.delete(self.keyname)

    def update_to_db (self):
        """
        Updates the current state of the entity from memcache to the datastore
        """
        self._db_version = self._cache_version
        logging.info('About to write into db. Key: %s' %self.keyname)
        self.update_cache ()
        return super (GlobalVersionedCachingModel, self).put ()

    def update_cache (self):
        """
        Updates the memacahe for this entity
        """
        memcache.set (self.keyname, serialize_entities (self))

    def put (self):
        self._cache_version += 1
        memcache.set (self.keyname, serialize_entities (self))
        if self._cache_version - self._db_version >= self._fault_tolerance or \
                                                self._cache_version == 1:
            self.update_to_db ()

    def delete (self):
        self.remove_from_cache()
        return super (GlobalVersionedCachingModel, self).delete ()

    @classmethod
    def get_by_key_name (cls, key_names, parent=None, **kwargs):
        try:
            parent = db._coerce_to_key (parent)
        except db.BadKeyError, e:
            raise db.BadArgumentError (str (e))
        rpc = datastore.GetRpcFromKwargs (kwargs)
        key_names, multiple = datastore.NormalizeAndTypeCheck (key_names, basestring)
        logging.info(key_names)
        keys = [datastore.Key.from_path (cls.kind (), name, parent=parent) for name in key_names]
        if multiple:
            return get2 (keys)
        else:
            return get2 (keys[0], rpc=rpc)

Similar Readings

• Models Caching at App Engine CookBook
• This article at Cook Book - http://appengine-cookbook.appspot.com/recipe/entity-caching-with-internal-version-for-high-readwrite-ops/

A friend of mine asked me to create a python script which checks if the mysql server is running or not. So, I could come up with this simple script. Although, there are many script available to do the same, in probably better manner, but still here it is. Hope you find it useful.

Resources

1. Repo: http://code.google.com/p/pranav/source/browse/poll_mysql/polling.py