Discussion: Method descriptors and fix attribute lookup order

Open

jahebaopened 8d ago

The problem

In Python there are three kind of method calls: regular methods, class methods and static methods.

Consider invoking method m on qualifier q in q.m(...).

If q is an object:

m is a regular method: m is called as m(q, ...)
m is class method: m is called as m(type(q), ...)
m static method: m is called as m(...)

In addition, if m is a regular attribute on object q, it is just called as m(...).

However, if q is a class:

m is a regular method: m is called as m(...)
m is class method: m is called as m(q, ...)
m static method: m is called as m(...)

CPython solves this through descriptors: most callables expose __get__ that allows them to act as a method:

def eq(self, arg):
    return self == arg

class X(int):
    pass

X.eq = eq

assert X(42).eq(42)

Thus, we can also get a bound method by calling __get__ directly:

x = X(42)
bound_method_eq = eq.__get__(x, X)

assert bound_method_eq(42)

Therefore, we can re-implement regular, class and static methods ourselves:

from functools import partial

class BaseMethod:
    def __init__(self, fn):
        self.fn = fn

class Method(BaseMethod):
    def __get__(self, obj, cls):
        if obj is None:
            return self.fn

        return partial(self.fn, obj)

class ClassMethod(BaseMethod):
    def __get__(self, obj, cls):
        if obj is not None:
            cls = type(obj)

        return partial(self.fn, cls)

class StaticMethod(BaseMethod):
    def __get__(self, obj, cls):
        return self.fn

class X(int):
    @Method
    def eq(self, other):
        return self == other

    @ClassMethod
    def doubled(cls, value):
        return cls(value + value)

    @StaticMethod
    def zero():
        return 0

one = X(1)
assert one.eq(1)
assert X.eq(1, 1)

assert one.doubled(2) == 4
assert X.doubled(2) == 4

assert one.zero() == 0
assert X.zero() == 0

Descriptor lookup works differently on objects and classes:


def get_attr(q, name):
    if type(q) is type:
        return type_get_attr(q, name)

    return object_get_attr(q, name)

def object_get_attr(obj, name):
    if name in obj.__dict__:
        return obj.__dict__[name]

    cls = obj.__class__

    if name in cls.__dict__:
        value = cls.__dict__[name]
        if hasattr(value, '__get__'):
            return value.__get__(obj, cls)
        return value
        
    raise AttributeError(name)


def type_get_attr(cls, name):
    if name in cls.__dict__:
        value = cls.__dict__[name]
        if hasattr(value, '__get__'):
            return value.__get__(None, cls)
        return value
        
    raise AttributeError(name)

Now we can do:

def invoke(q, n, *args):
    return get_attr(q, n)(*args)

assert invoke(one, "eq", 1)
assert invoke(X, "eq", 1, 1)

assert invoke(one, "doubled", 2) == 4
assert invoke(X, "doubled", 2) == 4

assert invoke(X, "zero") == 0
assert invoke(one, "zero") == 0

The above works, but is costly. Implementing method calls as actual descriptor lookups introduces a lot of unneeded overhead. Instead, we can do:


def invoke_direct(q, n, *args):
    # q is class
    if type(q) is type:
        m = q.__dict__[n]

        if isinstance(m, ClassMethod):
            return m.fn(q, *args)

        if isinstance(m, (StaticMethod, Method)):
            return m.fn(*args)
    
        return m(*args)

    # q is obj
    if n in q.__dict__:
        return q.__dict__[n](*args)

    c = q.__class__
    m = c.__dict__[n]

    if isinstance(m, Method):
        return m.fn(q, *args)

    if isinstance(m, ClassMethod):
        return m.fn(c, *args)

    if isinstance(m, StaticMethod):
        return m.fn(*args)

    return m(*args)

More importantly, we can actually divide the invocation into two steps:

def prepare(q, n):
    # q is class
    if type(q) is type:
        m = q.__dict__[n]

        if isinstance(m, ClassMethod):
            return m.fn, "self"

        if isinstance(m, (StaticMethod, Method)):
            return m.fn, "none"
    
        return m, "none"

    # q is obj
    if n in q.__dict__:
        return q.__dict__[n], "none"

    c = q.__class__
    m = c.__dict__[n]

    if isinstance(m, Method):
        return m.fn, "self"

    if isinstance(m, ClassMethod):
        return m.fn, "cls"

    if isinstance(m, StaticMethod):
        return m.fn, "none"

    return m, "none"

def execute(fn, how, q, *args):
    if how == "none":
        return fn(*args)
        
    if how == "self":
        return fn(q, *args)

    if how == "cls":
        return fn(q.__class__, *args)

def invoke_prepare(q, n, *args):
    fn, how = prepare(q, n)
    return execute(fn, how, q, *args)


assert invoke_prepare(one, "eq", 1)
assert invoke_prepare(X, "eq", 1, 1)

assert invoke_prepare(one, "doubled", 2) == 4
assert invoke_prepare(X, "doubled", 2) == 4

assert invoke_prepare(X, "zero") == 0
assert invoke_prepare(one, "zero") == 0

For native types, this can even be simplified further. For example, we know that int objects don't have custom attributes, so there is no need to lookup (1).__dict__.

Bringing this to Monty.

Monty currently only has a CallAttr family of opcodes, which assumes that virtually all method calls are regular method calls.

There is also no support for descriptor, class or static methods.

Yet, for the first version, we might not need actual descriptors yet. #1089 introduces a notion of explicit regular and class methods.

However, we should address #976 and fix looking up the method before evaluating arguments.

This requires us to introduce new opcodes.

Given

q.m(...)

it turns into opcodes:

Load 'q'
PrepareMethodCall  'm'
...
MethodCall

PrepareMethodCall will look up m on q and return the instructions how to perform the actual method call. If m can't be found, an error is thrown before the arguments are evaluated.

PrepareMethodCall pushes two values on the stack: Callable and Receiver. If there is no receiver Value::Undefined is used.

Similarly, GetAttr can return a combined heap allocated tuple of callable using the existing BoundMethod.


Note, `Callable` is not guaranteed to be actually callable:

```py
class X:
    x = classmethod(42)

X.x # this is fine
X.x() # TypeError: 'int' object is not callable

Activity

No activity yet.