The Meme

People have probably seen this meme before with Python just riding on C. While the meme is true about how under the hood Python is just C, it has huge advantages for the language most people don't utilise.

What are C Extensions?

C extensions are modules that can be imported into Python that utilise C. This allows the code to run low level operations which Python does not allow.

In order to make a C extension module for Python you need to start with the C file. As an example we will just be making an addition function in C. For this we are going to have to set a few things, first is a #define which is PY_SSIZE_T_CLEAN. The reason we need to set this is due to Python objects being too big to represent the size in standard C 32-bit int on a 64-bit system.

Now like most C programs, we need to #include something and in this case it's Python.h. The reason we need this is it adds PyObjects, which is the structure used for everything from and to Python in C.

Using a variable in a C extension functions is very different from both languages as when they are passed they are turned into a struct with the value inside.

To start our function we will create a static function which returns a pointer to a PyObject. The function also needs a name which will be py_add. Now we need inputs for the function, which is always two no more no less. To start like in normal Python we need the self variable which is a pointer to a PyObject. We then have args which are the real arguments passed in by Python which are represented as a tuple that is a PyObject pointer.

Creating the function was easy now we need to write the code inside it. To start we need to convert the values from the args into C data types. We do this by creating two ints A and B which are the two values we will add together. If you are doing more complex code once you have converted objects to normal C data types you can just pass them like normal.

Converting our args into C data types requires us to parse them. When parsing make sure to have an if false return null, so it stops the function which means the unassigned variables are not used. For this you create an if PyArg_ParseTuple is false then return null as the code has been passed something which is not an int. For the PyArg_ParseTuple you put the pointer to the PyObject args then tell it that the first two should be ints by doing "ii" then you have it enter the values which are pointers as the memory addresses for A and B.

Now that we have converted our numbers into C data types we can use the C add operations as normal to add A and B together and put the output in sum. In order to return this value to Python we need to convert it from a C data type to a PyObject for an int this is done by doing return PyLong_FromLong(sum);

While you may think that we are done. We have just created the full function. Now need to add it to the list of methods for our Python module.

To add our function to our Python module we need to create an array which contains our functions. For this we create a static PyMethodDef array, this contains custom_methods inside of this array we have the functions we are adding to Python.

The custom py_add is added by creating an entry which starts with the name of the method in Python for this we will call it add then we need the C function which is called for us this is py_add after that we tell Python how the arguments are passed for us this is METH_VARARGS after that we set the Doc strings.

For this array we also need to tell Python where the end of the array is we do this by adding a stop sign, which is {NULL, NULL, 0, NULL}.

Packaging the module we made is the last step, it's done by creating a static struct using PyModuleDef as a blueprint called mymodule. Inside of the struct we have 5 things we need to set.

First is the base which sets the CPython boilerplate for us this is mandatory and the only option is PyModuleDef_HEAD_INIT.

Second we need to set the name of our module which is what we use to import it into Python for this I named it Cadd.

Third we have the docs for this module for this I just put "Addition in C". After that we have our memory size per Python interpreter for most people you just want to set this to -1 which is what I have it set to.

The last field we enter is where we enter the list of functions to become methods for us this is the custom_methods we created earlier.

Now we just need to add the function which is called when Python calls this module. For this, we create a function which is named PyInit_Cadd which has a return of PyMODINIT_FUNC and takes a void as input. Inside of the function you return a PyModule_Create(&mymodule).

Now we need to write a Python file to create the full module. In this file we need to import Extension and setup for setuptools. Then we need to define the C module, this is done by setting module to extension with the inputs of name which is Cadd for us and the sources which is the C file we made.

We need the setup script which has 4 fields name, version, description, and modules which is the module we created before.

Now to compile this we run:

python setup.py build_ext --inplace

That should create a .so or .pyd file which can just be used like it is written in Python if it's in the same folder.

import Cadd

print(Cadd.add(15, 88))

When should you use C Extensions?

Now that I have gone over how to create a C extension for Python when should you do it. For me a big thing is when you develop a project which is getting CPU or memory bottlenecked, by a specific function or task. this is where importing from a C module is a drop in replacement with increased performance.

Combining the best of both worlds

My favourite thing about Python is its speed of development which when you're doing rapid prototyping is an amazing thing. Some issues can happen when your side project takes off and you're serving a ton of users and you're doing computing for them it can be quite expensive. But a full rewrite in another language would be even more expensive.

If you're able to figure out what part of your Python project is slow and there is no better way to optimise your code C modules are the perfect solution as it's a drop in replacement, While providing a performance uplift.

If you want to download the full code get it here: GitHub.