Skip to content

Provide a kcachegrind viewer and update docs #290

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mattip merged 2 commits into from
Oct 8, 2026
Merged

Provide a kcachegrind viewer and update docs #290

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
747bca6
add a kcachegrind-compatible profile printer
mattip Oct 8, 2026
d84537f
rework documentation
mattip Oct 8, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
94 changes: 79 additions & 15 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,37 +1,97 @@
# VMProf Python package

[![Build Status on TravisCI](https://travis-ci.org/vmprof/vmprof-python.svg?branch=master)](https://travis-ci.org/vmprof/vmprof-python)
[![Build Status on TeamCity](https://teamcity.jetbrains.com/app/rest/builds/buildType:(id:VMprofPython_TestsPy27Win)/statusIcon.svg)](https://teamcity.jetbrains.com/project.html?projectId=VMprofPython)
[![Tests](https://github.com/vmprof/vmprof-python/actions/workflows/tests.yml/badge.svg)](https://github.com/vmprof/vmprof-python/actions/workflows/tests.yml)
[![Wheels](https://github.com/vmprof/vmprof-python/actions/workflows/cibuildwheel.yml/badge.svg)](https://github.com/vmprof/vmprof-python/actions/workflows/cibuildwheel.yml)
[![Read The Docs](https://readthedocs.org/projects/vmprof/badge/?version=latest)](https://vmprof.readthedocs.org/en/latest/)
[![Build Status on AppVeyor](https://ci.appveyor.com/api/projects/status/github/vmprof/vmprof-python?branch=master&svg=true)](https://ci.appveyor.com/project/planrich/vmprof-python)

**VMProf** is a lightweight statistical profiler for CPython and PyPy. It samples
the call stack of a running program and writes a profile file you can open in
several viewers.

Head over to https://vmprof.readthedocs.org for more info!

## Installation

```console
pip install vmprof
python -m vmprof <your program> <your program args>
```

Our build system ships wheels to PyPI (Linux, Mac OS X). If you build from source you need
to install CPython development headers and libunwind headers (on Linux only).
On Windows this means you need Microsoft Visual C++ Compiler for your Python version.
VMProf 0.6 supports CPython 3.10 through 3.14 and PyPy, on Linux, Mac OS X and
Windows. Native profiling is available on Linux and Mac OS X.

Wheels are published to PyPI for all three platforms with libunwind bundled in.
If you build from source you need the CPython development headers, and on Linux
the libunwind headers as well — on Debian or Ubuntu, `python3-dev` and
`libunwind-dev`. On Windows you need the Microsoft Visual C++ Compiler for your
Python version.

## Quick start

Record a profile:

```console
$ python -m vmprof -o profile.prof <your program> <your program args>
```

Then open `profile.prof` in whichever viewer fits the question you're asking:

| Viewer | Good for | How |
| --- | --- | --- |
| `vmprofshow` | a quick look, no extra installs | `vmprofshow profile.prof tree` |
| [Firefox Profiler](https://profiler.firefox.com) | flame graph, timeline | `python -m vmprofconvert -convert profile.prof` |
| [kcachegrind](https://kcachegrind.github.io/) | callers/callees, call graph | `vmprofshow profile.prof callgrind -o profile.callgrind` |

Running `python -m vmprof` without `-o` prints basic statistics and keeps no
file.

### Firefox Profiler

The [vmprof-firefox-converter](https://github.com/Cskorpion/vmprof-firefox-converter)
converts a profile into a format the Firefox Profiler UI reads, giving you a
flame graph, a stack chart over time and an inverted call tree in the browser.
It understands PyPy's JIT frames too — see
[the announcement post](https://pypy.org/posts/2024/05/vmprof-firefox-converter.html)
for a tour.

```console
$ python -m pip install vmprof-firefox-converter
$ python -m vmprofconvert -convert profile.prof
```

### kcachegrind

`vmprofshow` can write the profile in callgrind format, which kcachegrind (or
`qcachegrind` on Mac OS X and Windows) reads:

```console
$ vmprofshow profile.prof callgrind -o profile.callgrind
$ kcachegrind profile.callgrind
```

The exported event is `Periods`: each sample is weighted by the time since the
previous one, in units of the sampling period, so costs are proportional to time
spent. At the default ~1kHz one unit is about 0.99ms.

Since vmprof samples the stack rather than instrumenting calls, it has no call
counts — every call edge is written as `calls=1`, so ignore kcachegrind's call
count column. Self cost is attributed to the line a function is defined on; use
`vmprofshow profile.prof lines` when you need line-level numbers.

## Development

Setting up development can be done using the following commands:

$ virtualenv -p /usr/bin/python3 vmprof3
$ python3 -m venv vmprof3
$ source vmprof3/bin/activate
$ pip install meson-python meson ninja
$ pip install --no-build-isolation --editable .

You need to install python development packages. In case of e.g. Debian or Ubuntu the package you need is `python3-dev` and `libunwind-dev`.
Now it is time to write a test and implement your feature. If you want
your changes to affect vmprof.com, head over to
https://github.com/vmprof/vmprof-server and follow the setup instructions.

Run the tests with:

$ pip install pytest cffi setuptools
$ python -m pytest vmprof/

Consult our section for development at https://vmprof.readthedocs.org for more
information.
Expand Down Expand Up @@ -128,7 +188,7 @@ helpful when functions exist that get called from multiple places, where each
invocation does not consume much time, but all invocations taken together do
amount to a substantial cost.
```console
$ vmprofshow vmprof_cpuburn.dat flat andreask_work@dunkel 15:24
$ vmprofshow vmprof_cpuburn.dat flat
28.895% - _PyFunction_Vectorcall:/home/conda/feedstock_root/build_artifacts/python-split_1608956461873/work/Objects/call.c:389
18.076% - _iterate:cpuburn.py:20
17.298% - _next_rand:cpuburn.py:15
Expand All @@ -148,7 +208,7 @@ $ vmprofshow vmprof_cpuburn.dat flat
```
Sometimes it may be desirable to exclude "native" functions:
```console
$ vmprofshow vmprof_cpuburn.dat flat --no-native andreask_work@dunkel 15:27
$ vmprofshow vmprof_cpuburn.dat flat --no-native
53.191% - _next_rand:cpuburn.py:15
46.809% - _iterate:cpuburn.py:20
0.000% - test:cpuburn.py:36
Expand All @@ -159,8 +219,8 @@ functions called. (In `--no-native` mode, native-code callees remain included
in the total.)

Sometimes it may also be desirable to get timings *inclusive* of called functions:
```
$ vmprofshow vmprof_cpuburn.dat flat --include-callees andreask_work@dunkel 15:31
```console
$ vmprofshow vmprof_cpuburn.dat flat --include-callees
100.000% - <native symbol 0x7f0dce8cca80>:-:0
100.000% - test:cpuburn.py:36
100.000% - burn:cpuburn.py:27
Expand All @@ -179,3 +239,7 @@ $ vmprofshow vmprof_cpuburn.dat flat --include-callees
0.356% - <native symbol 0x563a5f4ed8f1>:/home/conda/feedstock_root/build_artifacts/python-split_1608956461873/work/Objects/longobject.c:3432
```
This view is quite similar to the "tree" view, minus the nesting.

### Callgrind output

See [kcachegrind](#kcachegrind) above.
19 changes: 0 additions & 19 deletions docs/data.rst
View file
Open in desktop

This file was deleted.

109 changes: 40 additions & 69 deletions docs/development.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,94 +1,65 @@
Develop VMProf
==============

VMProf consists of several projects working together:

* `vmprof-python`_: The PyPI package providing the command line interface to enable vmprof.
* `vmprof-server`_: Webservice hosted at `vmprof.com`_. Hosts and visualizes data uploaded by `vmprof-python`_ package.
* `vmprof-integration`_: Test suite for pulling together all different projects and ensuring that all play together nicely.
* `PyPy`_: A virtual machine for the Python programming language. Most notably it contains an implementation for the logging facility `vmprof-server`_ can display.

The following description helps you to set up a development environment on Linux. For Windows
and MacOSX the instructions might be similar.
vmprof is made up of a Python package and a C extension, built with
`meson-python`_. The `PyPy`_ side of the JIT log support lives in PyPy itself.

.. _`meson-python`: https://mesonbuild.com/meson-python/
.. _`PyPy`: http://pypy.org
.. _`vmprof.com`: http://vmprof.com
.. _`vmprof-python`: https://github.com/vmprof/vmprof-python
.. _`vmprof-server`: https://github.com/vmprof/vmprof-server
.. _`vmprof-integration`: https://github.com/vmprof/vmprof-integration

Develop VMProf on Linux
-----------------------

It is recommended to use Python 3.x for development. Here is a list of requirements
on your system:

* python
* sqlite3
* virtualenv

Please move you shell to the location you store your source code in and setup
a virtual environment::

$ virtualenv -p /usr/bin/python3 vmprof3
$ source vmprof3/bin/activate

All commands from now on assume you have the vmprof3 virutal environment enabled.

Clone the repositories
----------------------
Setting up
----------

::
Create a virtual environment and install vmprof in editable mode::

$ git clone git@github.com:vmprof/vmprof-integration.git
$ git clone git@github.com:vmprof/vmprof-server.git
$ git clone git@github.com:vmprof/vmprof-python.git
# on old mercurial version the following command takes ages. please use a recent version
$ hg clone ssh://hg@bitbucket.org/pypy/pypy # optional, only if you want to hack on pypy as well

VMProf Server
-------------

::

# setup django service
$ cd vmprof-server
$ pip install -r requirements/development.txt
$ python manage.py migrate
# to run the service
$ python manage.py runserver -v 3

VMProf Python
-------------

An optional stage. It is only necessary if you want to co develop `vmprof-python`_ with `vmprof-server`_::

# install vmprof for development (only needed if you want to co develop vmprof-python)
$ cd vmprof-python
$ python3 -m venv vmprof3
$ source vmprof3/bin/activate
$ pip install meson-python meson ninja
$ pip install --no-build-isolation --editable .

Because the build is ``--no-build-isolation``, the C extension is rebuilt on
import when you change anything under ``src/``, so there is no separate build
step while developing.

Now you are able to change both the python package and the server and see the results.
Here are some more hints on how to develop this platform
You need your distribution's Python development headers, and on Linux the
libunwind headers as well. On Debian or Ubuntu those are ``python3-dev`` and
``libunwind-dev``.

Smaller Profiles
----------------
Running the tests
-----------------

::

Some times it is tedious to generate a big log file and develop a new feature with it.
Both for VMProf and JitLog you can generate small log files that ease development.
$ pip install pytest cffi setuptools
$ python -m pytest vmprof/

There are small logs generated by a python script in `vmprof-server/vmlog/test/data/loggen.py`. Use the following command to load those::
Some tests build small C extensions to exercise native profiling, which is why
``cffi`` and a compiler are needed.

$ ./manage.py loaddata vmlog/test/fixtures.yaml
Smaller profiles
----------------

Now open your browser and redirect them to the jitlog. E.g. http://localhost:8000/#/1v1/traces
Reading a profile of a long run is tedious while working on a feature. The
``vmprof/test/`` directory holds small recorded profiles, and
``vmprof/test/cpuburn.py`` generates fresh ones::

Integration Tests
-----------------
$ python -m vmprof -o profile.prof vmprof/test/cpuburn.py

This is a very important test suite to ensure that all packages work together. It is automatically run every day by travis. You can run them locally. If you happen not to run a Debian base distribution, you can provide the following shell variable to prevent the tests from downloading a Debian PyPy::
Working on the output modes
---------------------------

$ TEST_PYPY_EXEC=/path/to/pypy py.test testvmprof/
The viewers in :doc:`viewers` all read the same profile file, so a profile
recorded once can be replayed through every mode while you iterate::

$ vmprofshow profile.prof tree
$ vmprofshow profile.prof flat
$ vmprofshow profile.prof callgrind -o profile.callgrind

The printers live in ``vmprof/show.py``. Each one subclasses
``AbstractPrinter`` and implements ``_show(tree)``, where ``tree`` is the
``Node`` tree built by ``Stats.get_tree()``; see ``vmprof/stats.py`` for what a
node carries. ``vmprof/test/test_show.py`` builds ``Node`` trees by hand, which
is the quickest way to test a new output mode without recording anything.
18 changes: 10 additions & 8 deletions docs/faq.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,22 +9,24 @@ Frequently Asked Questions

* **Is it possible to just profile a part of my program?**: Yes here an example how you could do just that::

with open('test.prof', 'w+b') as fd:
with open('profile.prof', 'w+b') as fd:
vmprof.enable(fd.fileno())
my_function_or_program()
vmprof.disable()

Upload it later to vmprof.com if you choose to inspect it further::
Then open ``profile.prof`` in any of the viewers described in
:doc:`viewers`.

$ python -m vmprof.upload test.prof



* **What do the colors on vmprof.com mean?**: For plain CPython there is no particular meaning, we might change
that in the future. For PyPy we have a color coding to show at which state the VM sampled (e.g. JIT, Warmup, ...).
* **Which viewer should I use?**: ``vmprofshow`` is bundled and needs nothing
installed, the Firefox Profiler gives you a flame graph and a timeline, and
kcachegrind gives you caller and callee lists and a call graph. See
:doc:`viewers`.

* **My Windows profile is malformed?**: Please ensure that you open the file in binary mode. Otherwise Windows
will transform ``\n`` to ``\r\n``.

* **Do I need to install libunwind?**: Usually not. We ship python wheels that bundle libunwind shared objects. If you install vmprof from source, then you need to install the development headers of your distribution. OSX ships libunwind per default. If your pip version is really old it does not pull wheels and it will end up compiling from source.

* **Why are the call counts in kcachegrind all 1?**: Because vmprof samples
the stack rather than instrumenting calls, so it never sees an individual
call and cannot count them. See :doc:`viewers`.
23 changes: 13 additions & 10 deletions docs/index.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,30 +7,33 @@
|
|

VMProf Platform
===============
vmprof
======

`vmprof`_ is a platform to understand and resolve performance bottlenecks in your code.
It includes a *lightweight profiler* for `CPython`_ 2.7, `CPython`_ 3 and `PyPy`_
and an assembler log visualizer for `PyPy`_. Currently we support Linux, Mac OS X and Windows.
`vmprof`_ is a lightweight `statistical profiler`_ for `CPython`_ 3.10+ and
`PyPy`_, along with an assembler log reader for `PyPy`_. It runs on Linux,
Mac OS X and Windows.

The following provides more information about CPU profiles and JIT Compiler Logs:
Profiling writes a profile file, which you open in the viewer of your choice:
the bundled ``vmprofshow``, the Firefox Profiler, or kcachegrind::

pip install vmprof
python -m vmprof -o profile.prof <program.py> <program arguments>
vmprofshow profile.prof tree

.. toctree::
:maxdepth: 2

vmprof
viewers
faq
development
native
format
jitlog
query
data
development

.. _`CPython`: http://python.org
.. _`PyPy`: http://pypy.org
.. _`vmprof`: https://github.com/vmprof/vmprof-python
.. _`statistical profiler`: https://en.wikipedia.org/wiki/Profiling_(computer_programming)#Statistical_profilers
.. _`gperftools`: https://code.google.com/p/gperftools/
.. _`vtune`: https://software.intel.com/en-us/intel-vtune-amplifier-xe
Loading
Loading