PyCon India 2010 Experience

PyCon India this year was held at M.S. Ramaiah Institute of Technology, Bangalore this year on 25th and 26th of Sept. PyCon India 2010 is the primary Python conference in India. A purely volunteer effort, it is being hosted for the second time in India, and has attracted some of the best Python developers in India and abroad.

This year, I had a chance to speak on “Building Scalable Apps Using Google App Engine”. It was an intermediate level talk for 45 minutes. The presentation slides are available at http://in.pycon.org/2010/talks/5-building-scalable-apps-using-google-app-engine

All the presentation slides and other details are available on PyCon India site at http://in.pycon.org/2010/talks. Videos for all the talks shall be made available on YouTube soon, possibly by this week. But could take more time as well. Below are a couple of talks which were quite impressive and should be useful to the developer community are : –

Introduction to Image processing in Python, Anil C R – http://in.pycon.org/2010/talks/33
Python + Wii = Intutive Control, Asim Mittal – http://in.pycon.org/2010/talks/22
Text Parsing in Python, Gayatri Nittala, Madhubala Vasireddy – http://in.pycon.org/2010/talks/57
Rapid 3D game development in Blender, Arun Ravindran – http://in.pycon.org/2010/talks/37
Into 3D with VPython, Deepak Mishra – http://in.pycon.org/2010/talks/81
Agent Based User Simulation and Load Testing, Om Narayan – http://in.pycon.org/2010/talks/41
Internet Protocols in the Python Standard Library, Senthil Kumaran – http://in.pycon.org/2010/talks/14
Building Scalable Apps using Google App Engine, Pranav Prakash – http://in.pycon.org/2010/talks/5
Python Object Model and Introduction to meta programming, Sambasiva S – http://in.pycon.org/2010/talks/13
Test Driven Development, Gayatri Nittala, Madhubala Vasireddy – http://in.pycon.org/2010/talks/58
Python on Mobile Phones, N.Seshadri – http://in.pycon.org/2010/talks/34

I personally could not attend all the talks mentioned above, as some were conflicting with others, but I am looking forward to watching their YouTube developer videos. I will be updating you all when the videos are available.

Also, I had a chance to meet a lot of exiting and energrtic particles – Noufal Ibrahim, Anand Pillai, Abhishek Mishra (@ideamonk) and many more.

I am already looking forward for PyCon India 2011.

And yeah, the food was simply awesome. I really liked it 🙂

Preparing for the talk in PyCon India 2010

So, my talk on ‘Building Scalable Apps using AppEngine‘ in PyCon India 2010 has been approved. Although I have mentioned in the talk details as to how will my talk proceed, I would like to elaborate more on this.

Building Scalable Apps using Google App Engine (Python) (45 minutes )

Now there are a couple of keywords that must be understood clearly. We will proceed one by one on it.

Building Scalable Apps

So, we will be talking about building scalable applications. Building, as in starting from scratch. I will be presuming that the audience has some knowledge of web application development using a framework in any programming language. However acquaintance with Django, TurboGears is highly preferred. I am not expecting a lot of people to be used with Google App Engine, so I will be talking about Google App Engine in some details. What is Google App Engine, What is the purpose of it’s existence, what are its chief features, how is it different from other web application development framework. Then we will be diving into the greatest advantage of Google App Engine over others – Scalability. In some of the talks that I have taken on App Engine, I have seen developers with a blurred view on Scalability. Most of the times, people are mistaken for what in true sense is scaling of application/infrastructure. Scaling is so important to understand because we have been living a world where most of the things we see around are non scalable. Look around yourself, we have the same roads that existed years ago but traffic has been growing exponentially. The cities are the same, railways are the same and so is everything else. It is getting day by day difficult to sustain the burden we put on our limited resources. But nature and we do scale to certain limits. You heart beats normally at 72 beats per minute. While you are running or playing, your body needs more oxygen and so your heart stats to pump at a faster rate in order to cope up with the increased demand. This is called as scaling up. On the other hand consider your brain. Whatever we see, sense, smell are stored in neurons as patterns. The things we need to recall more frequently are stored at much larger neural networks as compared to things that we need infrequently. This is called as scaling out. I hope this explains the difference between a non-scalable architecture, a scale-up architecture and a scale-out architecture. Having understood that, we must know that Google App Engine is designed to scale out. This means that Google App Engine is very efficient at performing tasks at parallel. And this is what we must keep in mind while developing applications on App Engine.

Using Google App Engine

What differentiates App Engine is that it provides us the same infrastructure that is used by Google. Apart from many other things, it implies that we will be using Google BigTable for storing data. AppEngine provides a DataStore environment that is build on top of BigTable. DataStore is not SQL, it is not FlatFile, it is not RDBMS. It is BigTable. And it has its own rules, own ways to read/write from datastore, own ways to maintain concurrency and all other things that should be there in a modern database system. And there are certain restrictions, certain flexibility onto it. In order to be able to develop a scalable application using App Engine, it is very necessary that we know what is datastore capable of and what is outside the limits of datastore. In any application, there are more reads as compared to writes, so we need to make sure that our read latency is very low. And here comes the concept of write once, read many. We take care of all the possible ways in which our data might be needed to read, and we store them at the write time only. We also need to take care in how we write data as there are again a lot of overheads involved in Index building, searching, and quering. Apart from all these things we can do with datastore, we have a high speed, in memory cache called as ‘Memcache’. Used with a perfect blend of Datastore and memcache, the application can be quite fast and scalable.

I am also working on gathering the measurable statistics for a scalable code and a non-scalable code. I hope to complete it by this weekend, and i will upload it here soon.

Apart from this, I will be talking about a little technical background of Oxylabs Networks, India’s leading social apps and game developing company. How we began with MySQL, where MySQL lagged and why we chose AppEngine. How we cater to the needs of our more than 20 million player base with high scalability using App Engine.

And if all of this excites you, creates a chill down your spine, we are looking forward to give you an opportunity to be a part of our team. Attend the talk for more details on this.

PyCon India 2010,
25th and 26th Sept, 2010
M S Ramaiah Institute of Technology, Bangalore, India (map)

Update

Presentation Slide for PyCon India is shared here

My journey to Python

It starts even before one can think. So, i was in my first semester of my Bachelor’s in Engineering and it was December of 2004. I was back at my home place, Jaipur for vacations. There i met my school friend Himanshu Bhojwani, who made some real cool robotics and other electronics gadgets and he showed me them. I was quite curious and asked him how did he do all that stuf. Technology and gadgets had always interested me. So, he told me about Parallel Port programming and about Visual Basic 6.0. He said that he did all this by connecting his electronics stuff to parallel port of his computer and use Visual Basic to get his stuff done. That evening while returning back home, i purchased Parallel Port Programming. My initial plan was to use C++ to do the programming part, but looking at the ease VB6.0 could make same programs in much less time, i decided to learn VB6.0. I also got a neighbor of mine, who was a Software Engineer at that time, to guide me out with VB. Well, as time went on, i forgot the Parallel port programming, and was making small applications in VB6.0. This language opened a complete new world for me. From C++ to VB 6.0.

It dates back to the Jan of 2006. I was a fourth semester student student at Bharati Vidyapeeth University, Pune and those were the days of college tech-fest, Bharatiyam. I was in the college technical team, softwares division, supervised by Hemant Sahni, who was technical head at that time. He is real good when it comes to application software development. I was initially working with the networking team, but when he found out that i had good VB skills, he asked me to help him, as he was using VB and SQL Server 2000 for this application he was designing. We were making a software to manage event scheduling, registrations, accounts and other details for the techfest. At the same time, another technical head of our University, Siddharth Upmanyu was working on a online c/c++ compiler. (Yea, the same which was hosted earlier on this blog). Somehow i was quite interested in this “online compiler” business. So, i went to him, and asked ‘Sir, how have you made this cool stuff?’. And he gave me a detailed information about the underlying architecture and flow, most of which bounced off my brains. So i went ahead with another question, ‘Which language you used to code all this stuff?’. ‘Perl’, came the reply. I had no clue there was a programming language “Perl”. What kind of a name is this, i was thinking. And then i said, “Perl?”. Judging my expressions, he said “its a language similar to python in some aspects”. And i was even more confused. How can anyone name a language as “Python”. Obviously i had no clue what Python was. But I decided not to make another fool of myself, so i said “Oh Python! Its a cool language” and hurried back home. LoL

After i came back, i did a quick Google for Perl and Python, and decided to learn one of the language. Perl looked a little complex to me, so i went on for Python, and haven’t looked back since then. I used Python for almost everything i had to do programatically!! I had always been in love with Python.

After graduation, I joined Harbinger-Systems, Pune and was working on ColdFusion MX7 and SQl Server 2000. I used Python for some small programs and bot, which helped me in my day to day job. After that i am currently working with Oxylabs Networks, where I am working in a language i love, Python 🙂

This is in short, my love story with Python, a tale of inspiration, aspiration and willingness.

Is it possible to get users email by hosting gadget on appspot?

Google App Engine provides its User API service, using which you can allow people with Google Accounts to easily access your application. It works in a manner much similar to Google’s other login services. When you login, you are taken to a central page, common for all google services and then after you have authenticated yourself, you are redirected back to the service.

Since Google uses a central page for login and logout and it is a cookie based authentication, it might trigger question – if someone is logged into gmail or orkut or other such google service is it possible to get their Google Accounts using App Engine’s Users API? In order to do so, i deployed an application on App Engine, which would write “hello World!” on the canvas. This gadget XML is generated pragmatically and at the backend, there is a code which checks for a valid Google Account Holder.

The corresponding code goes as follows:
(There is no space at line 39, the space are mentioned here to go according to the syntax highlighter)

from google.appengine.api import users
from google.appengine.ext import webapp
from google.appengine.ext.webapp.util import run_wsgi_app
from google.appengine.ext import db

class Users(db.Model):
user = db.UserProperty()
created = db.DateTimeProperty(auto_now = True)
updated = db.DateTimeProperty(auto_now_add = True)

class Log(db.Model):
msg = db.TextProperty()
created = db.DateTimeProperty(auto_now=True)
level = db.StringProperty(default=’info’)

class MainPage(webapp.RequestHandler):
def get(self):
user = users.get_current_user()
if user:
Users.get_or_insert(‘u_’+user.nickname(), user=user)
else:
Log(msg=’User is None’).put()

self.response.headers[‘Content-Type’] = “application/xml”
self.response.out.write(“””

<![CDATA[

hello World!

] ] >

“””)

application = webapp.WSGIApplication([(‘/’, MainPage)], debug=True)

def main():
run_wsgi_app(application)

if __name__ == “__main__”:
main()

]]>

Upon deploying the gadget on orkut and gmail, i found that everytime the gadget loads, i just see a new entry in the Log, saying “User is None”.

This means that you can authenticate only users for your own domain/application, not for all of the Google products services 🙂

The complete source code of the application can be found at its Google Code repository http://code.google.com/p/codecontrol-samples/source/browse/user-authentication/src/.

Using Appstats in Glashammer

If you are using Google App Engine for developing and deploying your web application written in Python, chances are high that you must be using AppStats for measuring the performance of your application. Well, if you haven’t its the right time for you to start using it.
The details about how to use AppStats with Google App Engine in WebApp framework and the Django framework are available at the official app engine documentation.

Here, i will tell you how to use AppStats in your Glashammer application.

AppStats is a middleware, so it stands somewhere between your client and the app engine server. It measures each request coming and its response generated and provides valuable data for developers to tune up their app.

So here are the steps

Setup your application

Most probably, you might be performing setup of your application in the main.py. You need to add the middleware to the application. For this first import the appstats_wsgi_middleware class.
<![CDATA[
from google.appengine.ext.appstats.recording import appstats_wsgi_middlewareNext add the middleware where you are setting your application

app.add_middleware(appstats_wsgi_middleware)

Create a config file

The config file is a file named appengine_config.py located at the application root level. You can get a sample file from the location

google/appengine/ext/appstatus

in your SDK, and copy that file as appengine_config.py and save it in application root.

Adjust URL mappings in app.yaml

After you are done with these two you now need to map the URLs to the views of appstats view handler. In order to do this, modify your app.yaml file by adding the following line <![CDATA[- url: /stats.* script: $PYTHON_LIB/google/appengine/ext/appstats/ui.pyIf you have any mapping as /.*, make sure that you add the line above that mappings.

That’s it. You are done. Now when you run your application, you will see appstats details in the terminal log. You can also view the details for all prev records at /stats.

AtoZ-Help : version 0.2 released

The version 0.2 of AtoZ-help Framework is available for download and testing purposes. AtoZ-help is a framework written in Pure Python and runs on Google AppEngine for creating feature rich help pages online. The help pages can be integrated with your existing website or can be released as an independent site.

Check out the source code at : http://code.google.com/p/atoz-help/
Got a buF? Report it at : http://code.google.com/p/atoz-help/issues/list
The latest version is available for download. Download it here