kdevelop-python for python 3: beta release

Good news: Python 3.4 is about to be released, and with it kdevelop-python's first version to support Python 3. Until that happens in a few days, here's a beta:

kdev-python version 1.5.80-py3
http://download.kde.org/unstable/kdevelop/kdev-python/1.5.80/src/kdev-python-1.5.80-py3.tar.xz.mirrorlist
SHA256:99ca1ce97e2a7e553051be7505c17a921ab1aaf318999826ea285f771bcc538a

The kdev-python 1.6-py3 series is compatible with KDevelop 4.6 (kdevplatform 1.6) and is suitable for working with Python 3.x source code.
If you're only interested in using (as opposed to packaging or developing) kdev-python, you should consider installing kdev-python from your distribution's package manager instead of downloading the source code.

There are a few things which need to be announced about this, so read on!

Embedded Python fork

First and foremost, this abomination is finally gone in the -py3 series. Python 3.4 merged a patch which is required by kdev-python, and now we can use the system's python installation. This will especially make kdev-python comply with all distribution's requirements for packaging software, which means it will hopefully soon be available in all major distribution's package repositories.
This also means that kdev-python now depends on Python =3.4.

Python 2 compatibility

The py3 series is not compatible with Python 2, and currently you can not install kdev-python and kdev-python3 side-by-side. I will try to address this restriction in the future and I will also publish a script to install the two versions into separate environments, but for now that's what is like. Especially, for packagers, this means that kdev-python must conflict with kdev-python3. The problem cannot be solved by renaming all the files, it also requires new UI and glue code to be written to select the correct language version, which I only have done partially as of today.

Branch names in kdev-python.git

I also used this opportunity to reorganize the branches in kdev-python.git a bit:

• python3 is now python3-legacy (do not use)
• python3-nofork is now python3; this branch is the most recent (unstable) version of the python3 version of the plugin, and compiles against kdevplatform master
• there's a 1.6-py3 branch which is like 1.6 but for python 3.

If you had any of the renamed branches checked out, you might need to do git reset --hard origin/branchname. In the near future, master will also be renamed to python2 and python3 will be renamed to master (but not yet).

Feature comparison of kdev-python3 and kdev-python2

Generally, the Python 3 version has all the features from Python 2 plus a few more and some bug fixes. Not all new Python 3 features are understood yet (the syntax is supported, but the semantics isn't, e.g. nonlocal does nothing), though. What is there however is support for function annotations:

If the expressions in a function annotation represent a type (i.e. not an instance or something different) they are used to adjust the function's return and argument types.

I really hope this feature gets used for type hints, so this is a first step to encourage you to use it for that. ;)
Not all is awesome just yet, though: Although I'm not aware of any major issues, there will be regressions (things which were working in Python 2 but are now broken). There are always regressions, even with the ~90% test coverage kdev-python has. That's the purpose of this beta: go forth and test, and report all the bugs to the tracker so they can be slain!

KDevelop / Kate sprint in Barcelona in January 2014

Last week I have been in Barcelona at the KDevelop / Kate sprint with all the other nice people working on those projects. As always, it was very cool to meet everyone again and spend a week together improving software. A big thanks to the organizers and sponsors, too!

Since most of the time I work on a fairly encapuslated subsystem (Python support for KDevelop) and only a smaller part of my contributions goes towards the KDevelop / Kate core repositories, I don't  have any world-changing news to announce -- unlike for kate and KDevelop, no Qt5 / KF5 porting is happening yet in Python support. It doesn't make much sense to start porting until all the interfaces have stabilized.

Still, quite some interesting things have piled up in the last few months, and especially during the sprint.
In the following, I want to give a quick summary of those.

kdev-python 1.5 rc1 -- please test!

Hi!

Five months have passed since the release of kdev-python 1.4.1, and it's about time for the next release! The 1.5 release will contain various new features and bug fixes, and also will work with KDevelop 4.5, which is to be released soon.

One of the new features in kdev-python 1.5: initializing properties in class constructors via code completion

Notable Bugs fixed

The following issues were fixed, among others:

• 312275 don't add brackets if importing functions
• 318802 fix quadruple-quotes from causing completion to malfunction
• 317998 issue with function definition ranges causing completion to malfunction
• 317808 correct inheritance completion with member access
• 309817 correctly build uses for declarations in some corner cases
• 309469 include documentation files for PyKDE and the rest of PyQt
• 308986 fix a rare crash with some function definitions
• 1d34b03 fix an infinite loop in the debugger plugin
• 031badc don't reparse files on session restart if not necessary

The numbers are the bug / commit IDs of the change.

Notable new Features

The following new features are worth mentioning:

• 333cf91 sort completion items based on name similarity
• 6dc24ad class attribute initialization completion
• cd03cee PEP8 error checking. Enable it in "configure kdevelop" if you want to use it.
• b3ba512 support adding watches in the debugger plugin

Name-aware variable sorting: Although "the_bar" comes first alphabetically, "the_foo" is sorted first because its name matches the left side of the expression. You can still get an alphabetical list by invoking full completion.

Features that didn't make it into 1.5 yet

Two things that I wanted in 1.5 but which are not there are Python 3 support, and GHNS support. The latter is just not ready yet (might be in 1.5.1 or 1.5.2, but probably 1.6), the former can only be released in Q1 2014, as explained here.

Get it

kdev-python 1.5 is to be used with kdevplatform 1.5, and only 1.5 (so, kdevplatform with version numbers >= 1.4.60 and < 1.5.40). It's not going to work with kdevplatform master, or kdevplatform 1.4.
Please test this RC thoroughly and report any issues you find to the bugtracker, so the final 1.5 release can be as nice as possible! Get the tarball from download.kde.org. The sha256sum of the tarball is
441f675baddbfb9832a49458cc3c9a17daac4db55e7312c7a85d184bc0f4c776
Have fun with it! (not with the checksum, with the program I meant... you know)

kdev-python 1.4 stable released!

I'm happy to announce the release of the first stable version of kdev-python, version 1.4! As this is the first stable release, this post is supposed to be an overview about what kdev-python actually does.

KDevelop with kdev-python 1.4

First of all, kdev-python is a plugin for KDevelop. Its purpose is to make development of python applications more convenient. The main focus of the program is static analysis of source code, and providing features which use the information gathered, such as

• Semantic syntax highlighting (not the regex-stuff kate does, but real highlighting, showing your defined functions colorized etc.)
• Intelligent code completion, depending on where the cursor is and what variables exist etc.
• Navigation features, such as Jump to Declaration, searching for functions/classes, class browser, ...
• and many more

There's also a few features other than static code analysis, such as debugger integration, but the former is clearly the main focus of the project. kdev-python itself is written in C++.

Blender: exporting camera tracking markers to CSV

For university, I needed a camera tracker which could track motion of some particles under a microscope, and give me the coordinates of a particle for each frame. Blender has a very nice camera tracker which is ridiculously easy to use (and very fast), but it cannot export its results in a gnuplot-compatible format. But, it has a nice Python API, so you can just do it yourself! This is a script which exports all camera tracking markers of all videos in a blender file to CSV files in a directory called "data":

import bpy
D = bpy.data

for clip in D.movieclips:
    for track in clip.tracking.tracks:
        fn = 'data/tr_{0}_{1}.csv'.format(clip.name.split('.')[0], track.name)
        with open(fn, 'w') as f:
            frameno = 0
            while True:
                markerAtFrame = track.markers.find_frame(frameno)
                if not markerAtFrame:
                    break
                frameno += 1
                coords = markerAtFrame.co.xy
                f.write('{0} {1}\n'.format(coords[0], coords[1]))

Just paste this to a blender text editor window and press Alt+P to run it. You have to create a directory named "data" manually before that.

Careful: The coordinate output is in units of the video height / width. So if your video is not square sized, you need to multiply the output by the video clip dimensions (clip.size[0] and clip.size[1]).

Vienna kdev/kate sprint: First kdev-python beta release and python3 support

This is the third day of the KDevelop / kate sprint in Vienna, and it's really fun and productive so far! There's several new things related to kdev-python which I'd like to write something about.

Improved code completion in kdev-python and call for help

I recently rewrote most of kdev-python's code completion code, as it was a huge mess (it relied largely on regular expression matching, which just isn't powerful enough to do this properly). The result is less buggy, easier to maintain, and has unit tests (yay!). In the process, I also implemented quite a few features which I want to post a few screenshots of.

There's a second issue I want to talk about in this post: A major problem which remains for kdev-python is the documentation data, that is, the way the program gets information about external libraries, for example functions and classes in those libraries. I'd love to have some help here -- if you want to help improve kdev-python, this is a great point to start! There is no knowledge required about kdev-python's code base, or even C++ (all that is to do here can be done in python). Read more below in case you're interested.

Another note: kdev-python now has its own component on bugs.kde.org. Please report any issues you might encounter there!

Debugging support in kdev-python

There's now support for (graphically) debugging python applications from within KDevelop, like it works with C++.
The implementation relies on communicating with the default python debugger, pdb, which is shipped with python. It aims to nicely map all of pdb's features to buttons and tooltips / toolviews.

kdev-python debugger in action

About Python 3 support in kdev_python

Just a quick note about Python 3 support in the kdevelop python plugin: I will switch over to supporting Python 3 language features as soon as there's a release of the Django web framework using Python 3 (as this plugin was originally intended for developing Django applications, this seems like a reasonable point in time to me).

Most (all?) of the (syntactic) features Python 3 has so far are in Python2.7 anyways, so there's not much unsupported stuff. There's only one thing which really breaks compatibility with almost any script: the changes in the print() function (it was a statement in 2.7 but was made a function now, but with extra syntax sugar). Python 2 provides a way to emulate this behaviour; in the header of your file you need to put a line like

from __future__ import print_function

However, if you want to use kdev-python with your Python 3 project, you likely don't want to place those lines everywhere. That's why there's a new feature which allows you to specify a project-wide file header, which will be inserted into every python file in your project before it is parsed. This header file needs to be placed in your project root directory and needs to have the name ".kdev_python_header". You can, for example, put the above line in there to enable using the python 3 print function without syntax errors being thrown.

kdev-python: Argument type guessing, advanced list content type support and more

It has been a while since my last post about kdev-python, the python language support plugin for KDevelop; I have been quite busy recently. Neverthereless, I got quite a few cool features (almost) finished now.

Basic List content type support in KDevelop-Python

Hey,

after dealing with a few difficult problems, I finally got a first prototype of the "list content type support" implemented:

Doesn't look very impressive? It isn't, yet. But: The plugin now tracks which kind of data a list contains, and when accessing the list in one or another way, it uses that information to guess the type of the resulting objects. This is not very advanced yet, but it will be easy to extend now once the basics are there (and I think it will become very powerful if I can implement all the stuff I'm planning to). One important part of this feature will also be support for fixed-length lists (tuples), so you get correct types for code like

image_size = 800, 600
x, y = image_size # it should know that x, y are ints

Oh, and one more note: This feature is decorator-based, so all classes with a specific decorator are considered "container" classes. Thus, you can add your own containers by just using the @TypeContainer decorator, and the plugin will track their types.

Apart from adding that feature, I fixed some (more or less severe) bugs since alpha1.1, and the plugin can now parse pypy (one million LoC) without crashing (however it needs quite a few minutes to do so...).

Have Fun, and feel free to email me about problems.
Bye,
Sven

KDevelop python language support plugin pre-release

Hey there!

For the last couple of months, I have been working on Python language support for KDevelop, the well-known C++ IDE from the KDE project. Now I can finally release a first preview of my work to the public:

In this post I'll tell you about what the plugin is capable of and how to install it.