Talk

Dynamically generated methods with a non-generic signature

Thursday, May 23

11:00 - 11:30
RoomPanino
LanguageEnglish
Audience levelIntermediate
Elevator pitch

This talk teaches you how to use some python black magic to add methods to child classes of a parent class, based on certain properties of the child class. Introspection, descriptors, and __init_subclass__ are going to be our friends.

Abstract

In other words, Descriptors + PEP-362 (function signature object) and a seasoning of PEP-487 (simpler customization of class creation via __init_subclass__).

There are different ways to have generated methods and attributes attached to all classes in a library, and this talk presents the way we’re doing it in scikit-learn. Here you’ll understand the use-case, and see the details and challenges presented by it, and how we approached them.

This talk uses tools in the Python language which seem like black magic at first to most, but understanding them gives you very strong tools you can use in your daily work to customize your objects the way you need.

The use-case we study here goes as: we would like to add methods to all Estimators, which are all subclasses of the BaseEstimator. The signature of the methods generated depends on the signature of other methods existing in those subclasses, but we also want to give the option of modifying generated methods without having to change the existing methods.

The solution we present involves a few concepts which we’ll explain during the talk: Investigate the existing methods’ signature using inspect Manually traverse MRO (method resolution order) and inspect class attributes allowing for modifications on what the inspect has concluded Use a descriptor to generate methods accordingly Use PEP-362 to attach a signature object to the generated methods Dynamically generate docstrings for those methods Use PEP-487, aka __init_subclass__, to attach those methods to child classes when appropriate

This is a hands-on talk, explaining each concept in isolation and then showing how they fit together, and we’ll be presenting and testing code during the talk. You will be able to use any of the components used in this talk in your own work and fit them to what you’re doing.

TagsCPython
Participant

Adrin Jalali

Adrin is a scikit-learn maintainer and works on a few other open source projects. He has a PhD in Bioinformatics, has worked as a consultant, and in an algorithmic privacy and fairness team. He is now a cofounder at probabl.ai, where they work on enabling people to do statistically sane machine learning.