crocodile's blog

相对于decorator, iterator, generator, metaclass 这些概念，descriptor是python所有概念中最难理解的一个了。尽管python documentation对其描述不多(基本上是一笔带过)，以至于很多人忽略了这个概念。事实上，自己以前也忽略了它，只不过最近重温python时无意间发现了他，觉得非常有必要深入研究它。

自己的理解是建立在前人的基础之上的。不过，网上关于descriptor的资料实在是有限，下面是自己认为有参考价值的几个链接：
1.http://users.rcn.com/python/download/Descriptor.htm (最主要的参考资料)
2.http://www.python.org/peps/pep-0252.html (太难理解了)
3.http://www.python.org/~jeremy/weblog/030425.html
4.http://www.python.org/download/releases/2.2.2/descrintro/
5.Python25 documentation (Python Reference Manual, Chapter 3.4.2)
6.Python25的源代码

1.假如"x"是普通属性。
每个class均有一个__dict__，该class的obj也有一个__dict__。
1)对于obj.x，优先查找obj.__dict__["x"]，如果找不到，再查找class.__dict__["x"]；
2)对于class.x，直接查找class.__dict__["x"]；
3)obj.__dict__与class.__dict__可以拥有同名的属性；
4)obj.x=10，将无条件的更新obj.__dict__；class.x=10，将无条件的更新class.__dict__。

class C(object):
x = 10

b = C() #b.__dict__不包含"x"，C.__dict__包含"x"
b.x #10
C.x #10
b.x=100 #b.__dict__也包含一个"x"，对应的值为100；C.__dict__对应的值还是10
b.x #100
C.x #10
t=C()
t.x #10
C.x=99 #C.__dict__["x"]变成了99
t.x #99
b.x #100
C.x #99

2.假如"x"是一个descriptor
为简化，假设obj.__dict__没有"x"项。

class RevealAccess(object):
def __init__(self, initval, name):
self.val = initval
self.name = name

def __get__(self, obj, objtype):
print 'Retrieving', self.name
return self.val

def __set__(self, obj, val):
print 'Updating' , self.name
self.val = val

class C(object):
x = RevealAccess(10,"A test variable")


>>> b = C() # case1
>>> b.x
Retrieving A test variable
10
>>> C.x
Retrieving A test variable
10
>>>
>>> b.x=100 # case2
Updating A test variable
>>> b.x
Retrieving A test variable
100
>>> C.x
Retrieving A test variable
100
>>>
>>> C.x=1000 # case3
>>>
>>> b.x
1000
>>> C.x
1000
>>> # case4
>>> C.x = RevealAccess(999,"The second var")
>>> b.x
Retrieving The second var
999
>>> C.x
Retrieving The second var
999
>>>

上述结果可以通过Python25 Documentation (Python Reference Manual, Chapter 3.4.2.2)解释：
The following methods only apply when an instance of the class containing the method (a so-called descriptor class) appears in the class dictionary of another new-style class, known as the owner class. In the examples below, ``the attribute'' refers to the attribute whose name is the key of the property in the owner class' __dict__. Descriptors can only be implemented as new-style classes themselves.

1)__get__( self, instance, owner)
Called to get the attribute of the owner class (class attribute access) or of an instance of that class (instance attribute access). owner is always the owner class, while instance is the instance that the attribute was accessed through, or None when the attribute is accessed through the owner. This method should return the (computed) attribute value or raise an AttributeError exception.
(注意：obj.x、class.x均会导致该调用)

2)__set__( self, instance, value)
Called to set the attribute on an instance instance of the owner class to a new value, value.
(注意：仅obj.x=ttt均会导致该调用；class.x=ttt直接绑定到另外一个对象上)

3)__delete__( self, instance)
Called to delete the attribute on an instance instance of the owner class.


为简化以及方面理解，在实际应用中，尽量避免obj.__dict__与class.__dict__拥有同名的属性。


3.假如"x"是一个descriptor，而且obj.__dict__也有一个"x"项。非常复杂。
For instance bindings, the precedence of descriptor invocation depends on the which descriptor methods are defined. Data descriptors define both __get__() and __set__(). Non-data descriptors have just the __get__() method. Data descriptors always override a redefinition in an instance dictionary. In contrast, non-data descriptors can be overridden by instances.

详细的描述还得参考：http://users.rcn.com/python/download/Descriptor.htm

1)从该文档可以知道，descriptor与__getattribute__密切相关。基础类型object、type均定义了__getattribute__，并有缺省实现。上面所说的descriptor的各种特性其实就是这些__getattribute__的行为。也就是说，如果自己重新定义了__getattribute__，那么这个函数具有最高的优先级。

2)另外，__getattr__/__setattr__/__delattr__也与属性的取值有关，见Python25 documentation (Python Reference Manual, Chapter 3.4.2)。它们的级别比较低。建议它们所对应的属性不要为descriptor。

class RevealAccess(object):
def __init__(self, initval, name):
self.val = initval
self.name = name

def __get__(self, obj, objtype):
print 'Retrieving', self.name
return self.val


class C(object):
x = RevealAccess(10,"A test variable")
def __init__(self,val):
self.x = val

>>> b = C(100)
>>>
>>> C.x
Retrieving A test variable
10
>>> b.x
100
>>>


4.根据http://users.rcn.com/python/download/Descriptor.htm的描述，descriptor更多的被python自己用来实现一些语言上的特性。比如：function/property/staticmethod/classmethod/均是通过descriptor实现的。

其实主要是复习gcc的用法

#include <Python.h>

int main(void){
    Py_Initialize();
    PyRun_SimpleString("from time import time,ctime\n"
                        "print 'Today is',ctime(time()\n)");
    Py_Finalize();
    return 1;

}

假定python2.4安装在/opt/python目录下
gcc -I/opt/python/include/python2.4 -L/opt/python/lib -lpython2.4 -o test_py test_py.c

Python Performance Tips

This page is devoted to various tips and tricks that help improve the performance of your Python programs. Wherever the information comes from someone else, I've tried to identify the source.

If you have any light to shed on this subject, let me know.

Contents
Profiling Code
Profile Module
Trace Module
Sorting
String concatenation
Loops
Avoiding dots...
Local Variables
Initializing Dictionary Elements
Import Statement Overhead
Using map with Dictionaries
Data Aggregation
Doing Stuff Less Often
Profiling Code
The first step to speeding up your program is learning where the bottlenecks lie. It hardly makes sense to optimize code that is never executed or that already runs fast. I use two modules to help locate the hotspots in my code, profile and trace.

Profile Module
The profile module is included as a standard module in the Python distribution. Using it to profile the execution of a set of functions is quite easy. Suppose your main function is called main, takes no arguments and you want to execute it under the control of the profile module. In its simplest form you just execute

import profile
profile.run('main()')

When main() returns, the profile module will print a table of function calls and execution times. The output can be tweeked using the Stats class included with the module. For more details, check out the profile module's documentation (Lib/profile.doc).
Trace Module
The trace module is a spin-off of the profile module I wrote originally to perform some crude statement level test coverage. As of Python 2.0 you should find trace.py in the Tools/scripts directory of the Python distribution. Thanks to Andrew Dalke it's very easy to run from the command line to trace execution of whole scripts:

% trace.py -t spam.py eggs

There's no separate documentation, but there are some pretty complete docstrings at the top of the module.
Sorting
From Guido van Rossum

Sorting lists of basic Python objects is generally pretty efficient. The sort method for lists takes an optional comparison function as an argument that can be used to change the sorting behavior. This is quite convenient, though it can really slow down your sorts.

An alternative way to speed up sorts is to construct a list of tuples whose first element is a sort key that will sort properly using the default comparison, and whose second element is the original list element.

Suppose, for example, you have a list of tuples that you want to sort by the n-th field of each tuple. The following function will do that.

def sortby(list, n):
nlist = map(lambda x, n=n: (x[n], x), list)
nlist.sort()
return map(lambda (key, x): x, nlist)

Here's an example use:
>>> list = [(1, 2, 'def'), (2, -4, 'ghi'), (3, 6, 'abc')]
>>> list.sort()
>>> list
[(1, 2, 'def'), (2, -4, 'ghi'), (3, 6, 'abc')]
>>> sortby(list, 2)
[(3, 6, 'abc'), (1, 2, 'def'), (2, -4, 'ghi')]
>>> sortby(list, 1)
[(2, -4, 'ghi'), (1, 2, 'def'), (3, 6, 'abc')]

String Concatenation
Strings in Python are immutable. This fact frequently sneaks up and bites novice Python programmers on the rump. Immutability confers some advantages and disadvantages. In the plus column, strings can be used a keys in dictionaries and individual copies can be shared among multiple variable bindings. (Python automatically shares one- and two-character strings.) In the minus column, you can't say something like, "change all the 'a's to 'b's" in any given string. Instead, you have to create a new string with the desired properties. This continual copying can lead to significant inefficiencies in Python programs.

The following is adapted from inputs by Aaron Watters:

Avoid this:

str = ""
for substring in list:
str = str + substring

Use "".join(list) instead. The former is a very common and catastrophic mistake when building large strings. Similarly, if you are generating bits of a string sequentially instead of

str = ""
for x list:
str = str + some_function(x)

use
str = [None]*len(list):
for x in range(len(list):
str[x] = some_function(list[x])
str = "".join(str)

Avoid:

out = "" + head + prologue + query + tail + "

Instead, use
out = "%s%s%s%s" % (head, prologue, query, tail)

This is a lot faster, and easier to modify also. In addition, the slow way of doing things got slower in Python 2.0 with the addition of rich comparisons to the language. It now takes the Python virtual machine a lot longer to figure out how to concatenate to strings. (Don't forget that Python does all method lookup at runtime.)

Loops
Python supports a couple of looping constructs. The for statement is most commonly used. It loops over the elements of a sequence, assigning each to the loop variable. If the body of your loop is simple, the interpreter overhead of the for loop itself can be a substantial amount of the overhead. This is where the map function is handy. You can think of map as a for moved into C code. The only restriction is that the "loop body" of map must be a function call.

Here's a straightforward example. Instead of looping over a list of words and converting them to upper case:

newlist = []
for word in list:
newlist.append(word.upper())

you can use map to push the loop from the interpreter into compiled C code:
import string
newlist = map(string.upper, list)

List comprehensions were added to Python in version 2.0 as well. They provide a syntactically more compact way of writing the above for loop:

newlist = [s.upper() for s in list]

It's not really any faster than the for loop version, however.
Guido van Rossum wrote a much more detailed examination of loop optimization that is definitely worth reading.

Avoiding dots...
Suppose you can't use map? The example above of converting words in a list to upper case has another inefficiency. Both newlist.append and string.upper are function references that are recalculated each time through the loop. The original loop can be replaced with:
import string
upper = string.upper
newlist = []
append = newlist.append
for word in list:
append(upper(word))

Local Variables
The final speedup available to us for the non-map version of the for loop is to use local variables wherever possible. If the above loop is cast as a function, append and upper become local variables.

def func():
upper = string.upper
newlist = []
append = newlist.append
for word in words:
append(upper(word))
return newlist

An extra performance boost is received because local variables are accessed more efficiently than variables at module scope. On my machine (100MHz Pentium running BSDI), I got the following times for converting the list of words in /usr/share/dict/words (38,470 words) to upper case:

Version Time (seconds)
Basic loop 3.47
Eliminate dots 2.45
Local variable & no dots 1.79
Using map function 0.54

Eliminating the loop overhead by using map is often going to be the most efficient option. When the complexity of your loop precludes its use other techniques are available to speed up your loops, however.

Initializing Dictionary Elements
Suppose you are building a dictionary of word frequencies and you've already broken your words up into a list. You might execute something like:

wdict = {}
for word in words:
if not wdict.has_key(word): wdict[word] = 0
wdict[word] = wdict[word] + 1

Except for the first time, each time a word is seen the if statement's test fails. If you are counting a large number of words, many will probably occur multiple times. In a situation where the initialization of a value is only going to occur once and the augmentation of that value will occur many times it is cheaper to use a try statement:

wdict = {}
for word in words:
try:
wdict[word] = wdict[word] + 1
except KeyError:
wdict[word] = 1

It's important to catch the expected exception, and not have a default except clause to avoid trying to recover from an exception you really can't handle by the statement(s) in the try clause.

Note that if the try clause generates an exception most of the time, it will often be more efficient to test for the exceptional condition than to generate it and recover from it.

Import Statement Overhead
import statements can be executed just about anywhere. It's often useful to place them inside functions to restrict their visibility and/or reduce initial startup time. Although Python's interpreter is optimized to not import the same module multiple times, repeatedly executing an import statement can seriously affect performance in some circumstances.

Consider the following two snippets of code (originally from Greg McFarlane, I believe - I found it unattributed in a comp.lang.python/python-list@python.org posting and later attributed to him in another source):

def doit():
import string ###### import statement inside function
string.lower('Python')

for num in range(100000):
doit()

or:
import string ###### import statement outside function
def doit():
string.lower('Python')

for num in range(100000):
doit()

The second version will run substantially faster than the first, even though the reference to the string module is global in the second example. String methods were introduced to the language in Python 2.0. These provide a version that avoids import completely:
def doit():
'Python'.lower()

for num in range(100000):
doit()

The above example is obviously a bit contrived, but the general principle holds.

Using map with Dictionaries
I found it frustrating that to use map to eliminate simple for loops like:

dict = {}
nil = []
for x in list:
dict[x] = nil

I had to use a lambda form or define a named function that would probably negate any speedup I was getting by using map in the first place. I decided I needed some functions to allow me to set, get or delete dictionary keys and values en masse. I proposed a change to Python's dictionary object and used it for awhile. However, a more general solution appears in the form of the operator module in Python 1.4. Suppose you have a list and you want to eliminate its duplicates. Instead of the code above, you can execute:
dict = {}
map(operator.setitem, [dict]*len(list), list, [])
list = statedict.keys()

This moves the for loop into C where it executes much faster.
Data Aggregation
(Paraphrased from a note by Aaron Watters)

Function call overhead in Python is relatively high, especially compared with the execution speed of a builtin function. This strongly suggests that extension module functions should handle aggregates of data where possible. Here's a contrived example written in Python. (Just pretend the function was written in C. :-)

x = 0
def doit(i):
global x
x = x + i

list = range(10000)
for i in list:
doit(i)

vs.
x = 0
def doit(list):
global x
for i in list:
x = x + i

list = range(10000)
doit(list)

Even written in Python, the second example runs about four times faster than the first. Had doit been written in C the difference would likely have been even greater (exchanging a Python for loop for a C for loop as well as removing most of the function calls).
Doing Stuff Less Often
The Python interpreter performs some periodic checks. In particular, it decides whether or not to let another thread run and whether or not to run a pending call (typically a call established by a signal handler). Most of the time there's nothing to do, so performing these checks each pass around the interpreter loop can slow things down. There is a function in the sys module, setcheckinterval, which you can call to tell the interpreter how often to perform these periodic checks. In Python 1.4 it defaults to 10. If you aren't running with threads and you don't expect to be catching lots of signals, setting this to a larger value can improve the interpreter's performance, sometimes substantially.