Autocompletion in Python

This document describes how autocompletion works for Python modules, and explains the solution implemented in tudat to provide autocompletions for all Python and C++ modules (exposed through pybind11) in tudatpy.

Introduction

Autocompletion in Python is performed by a Python Language Server (VSCode uses such a server, and other IDEs as well), which works by scanning the current scope and using the active Python interpreter to gather information about installed packages and modules.

For this to work for a given module it must be inspectable. The tudat kernel, exposed to Python using pybind11 (kernel.so), is not.

Note

This is likely related to an issue with the def_submodule function of pybind11, due to which exposed modules are not fully inspectable by the Python interpreter (see [1]).

The solution that hs been implemented in tudatpy is to create a Python module that wraps each tudat kernel module, and explicitly imports all members of the kernel module. This way, the Python Language Server can inspect the module, and autocompletion works.

Goal of this guide

The goal of this guide is to give you immediate insight into how:

  1. We implement autocompletion for kernel modules in tudatpy

  2. How we use the expose_kernel_module.py [2] script to automate this process

  3. How you can use build_and_expose_kernel.sh [3] to simultaneously build tudatpy and expose all kernel modules (to automatically add autocompletion for any new kernel members you may have exposed)

Implementation in tudatpy

Autocompletion in tudatpy was implemented in pull request 128 [4].

To explain how this works, consider a tudat kernel module which is exposed through a Python module which wraps it. The Python module

  • lives inside tudatpy

  • has the same name as the kernel module

  • and has inside an __init__.py with the following statement: from tudatpy.kernel.<kernel_module_name> import * -this allows us to import all members of the kernel module from its Python wrapper-.

For each such exposed kernel module, this pull request does the following:

  1. Next to the Python module’s __init__.py, if and only if the module has non-module members (so classes, functions and other objects), a new file is created: _import_all_kernel_members.py

  2. _import_all_kernel_members.py explicitly imports all members (classes, functions and other objects) of the kernel module as follows: from tudatpy.kernel.<module_name> import <member 1>, <member 2>, ... (see [5])

  3. In the module’s __init__.py, we import everything from _import_all_kernel_members.py (see [6])

This way, all the explicitly imported kernel members are accessible from the Python module wrapping it: autocompletion engines will scan this, and detect all module members for autocompletion.

The reason we do this with two files (__init__.py and _import_all_kernel_members.py) is that this way users can change the __init__.py (which they may well want to do, as it is the __init__.py of a _hybrid_ C++/_Python_ module), and we can freely automatically generate the _import_all_kernel_members.py overwriting the existing ones, as these files will never be manually changed.

Basically, this way we allow users to change __init__.py, and keep the liberty to blast away old versions of all _import_all_kernel_members.py instances when the kernel changes.

How the process is automated

expose_kernel_module.py [2] automates this process. We will consider the numerical_simulation kernel module to explain how the script works. Given the name of the kernel module (numerical_simulation), expose_kernel_module.py [2] will:

  1. Expose (wrap) every submodule of numerical_simulation (create submodule directory and __init__.py file)

  2. For each submodule, create a _import_all_kernel_members.py file, which explicitly imports all non-module members of the submodule

Simultaneously compiling tudatpy, exposing all kernel modules and enabling autocompletions

The build_and_expose_kernel.sh [3] script automates the process of compiling tudatpy and exposing all kernel modules. It does the following:

  1. Builds tudatpy (by calling the build.sh [7], providing it with the number of cores to be used for compilation)

  2. Defines a list of kernel modules to be exposed (this is necessary as some kernel modules such as io and util are superseeded/wrapped by the tudatpy python modules io and utils)

  3. For each kernel module in the list, it calls expose_kernel_module.py [2] to expose the kernel module

References