Saturday, July 22, 2017

Expremigen: expressive midi generation


A lot of text-based midi specification languages exist, but those I know of suffer from one of the following drawbacks:
  • either the syntax is very low-level as in SKINI, which basically translates all midi events into text. This is great for round-tripping between midi and text, but it's not great for writing and interpreting "manually".
  • or the syntax makes it easy for specifying notes and rhythms but it lacks a way of easily specifying evolution of parameters over time (e.g. crescendos, rallentandos, rubatos, staccatos vs legato). These things are often left as "future work" or they feel like they are implemented as an afterthought and lack in expressive power.


I've been working with supercollider a lot lately, and one of its great features is an extensive pattern library. Patterns are like templates for generating music events. After thinking about how to solve the above problems in one easy to use language, I came to the conclusion that supercollider's patterns are an ideal foundation for building a solution.

After some searching I found the isobar library which implements supercollider's patterns in python 2. I noticed some drawbacks in isobar's approach though: it only supports python 2 at the moment, and their patterns appear to be using eager evaluation, meaning that you would not be able to specify patterns with very high ("infinite") repeat counts without overflowing your computer's memory space and exploding the required CPU time. Instead of porting isobar to python 3 I therefore decided to create a new implementation of of supercollider's patterns, based on python generators (i.e. using lazy evaluation), in the expremigen (expressive midi generation library) system.

The next step I took was to couple these patterns with the tweening possibilities found in my earlier vector animation library pyvectortween. By now I had a system that could flexibly animate midi properties like volumes over time.

While creating examples it became clear that writing code might scare off some potential users (especially if you are trying to convert existing music as opposed to generating new music algorithmically) and it was not close enough to my initial goal of creating a new midi dsl (dsl = domain specific language), so I set out to define a syntax and came up with a new midi specification language called MISPEL. MISPEL reuses the best ideas (imho) of  existing systems like lilypond and abc notation and then adds one of its own: property animations. For a more detailed introduction to MISPEL and some examples, please see the github readme file.

What does MISPEL look like? Where's the code?

Please see expremigen's github page for the complete source code of expremigen (Python 3 only, GPLv3 license) and some examples. This is of course a work in progress - despite quite some unit tests, bugs can still be expected, and things might change in the future.


The language is already quite capable, surpassing other systems in specific areas, and no doubt also lacking in other areas, but perhaps not feature complete yet. For now it concentrates on specifying notes, durations, and animatable properties like volume (crescendo, decrescendo), played duration (staccato, legato), lag (rubato) and tempo (accelerando, ritardando), midi control changes and pitchbend. Other things like program changes and sysex are not supported. Time will tell if and how this system needs to evolve.

One potentially interesting thing would be to somehow base my bluegrass music style compiler on mispel instead of lilypond. I expect this would quickly reveal some of the gaps in the language. 

Another interesting thing could be to investigate reverse engineering of midi files into MISPEL to use as a starting point for adding expressivity.

Tuesday, May 2, 2017

Mask driven vector animations with python


Mask images have many applications. In photo editing, masks e.g. are used to succinctly subscribe where to clip images, or to define regions where an image should be transparant. In video editing, masks are used to describe so-called wiping (fancy transitions from one image to the next). In 3d programs, there's a kind of 3d equivalent of masks, sometimes called dynamic paint, to describe influence of bones on vertices, or to dynamically create vertex colors and displacements.

Can we use mask images in combination with vector animation to do some cool stuff? I'd like to think we can. The idea is not to animate the mask image itself, but to interpret the values in the mask image as different animation parameters. E.g. consider the following example of a heart image. The red pixels are considered "masked" pixels, whereas the surrounding transparent pixels are considered unmasked values. Based on whether a pixel is masked or unmasked, we can render different animations. Let's apply the idea on an example to make it more concrete...

I'll use this wikimedia picture as input

And automagically turn it into the following pschychedelic animation


Well. Using the vectortween library and some gizeh and moviepy magic makes this relatively easy.
Here's the code. Sorry it's not particularly short or elegant, and it may need some tweaking with other mask images as input but it shows off the concept.

if __name__ == "__main__":
    import gizeh
    import moviepy.editor as mpy

    from scipy import misc
    from scipy.ndimage import zoom
    from numpy import pi
    from vectortween.NumberAnimation import NumberAnimation
    from vectortween.SequentialAnimation import SequentialAnimation
    import random

    # heart shape retrieved from https: // / wiki / File: Heart_coraz % C3 % B3n.svg    
    mask = misc.imread("heart.png")
    print("mask.shape = ", mask.shape, " mask.dtype = ", mask.dtype)
    H = mask.shape[0]
    W = mask.shape[1]
    subsample = 20    
    subsampled_mask = zoom(mask, (1 / subsample, 1 / subsample, 1))

    # debug code:    
    # misc.imsave("heart-subsampled.png", subsampled_mask)
    yrange = subsampled_mask.shape[0]
    xrange = subsampled_mask.shape[1]
    xwidth = subsample
    ywidth = subsample
    duration = 5    
    fps = 24

    radii = {}

    def get_color(x, y, subsampled_mask):
        color = subsampled_mask[y][x] / 255        
        return color

    def masked(x, y, subsampled_mask):
        c = get_color(x, y, subsampled_mask)
        if sum(c) > 2:
            return 1        
            return 0

    def make_frame(t):
        surface = gizeh.Surface(W, H)
        canim = SequentialAnimation([NumberAnimation(frm=0, to=2 * pi, tween=["easeOutQuad"])], repeats=int(duration))
        canim2 = SequentialAnimation([NumberAnimation(frm=2 * pi, to=0, tween=["easeOutQuad"])],
                                     repeats=int(2 * duration))
        radiusmod = SequentialAnimation([NumberAnimation(frm=0, to=10, tween=["easeInOutSine"]),
                                         NumberAnimation(frm=10, to=0, tween=["easeInOutSine"])],
                                        repeats=int(1.5 * duration))
        for y in range(yrange):
            for x in range(xrange):
                if masked(x, y, subsampled_mask):
                    pinkfactor = random.uniform(0.2, 0.8)
                    if (x, y) not in radii:
                        radii[(x, y)] = random.uniform(xwidth / 8, 3 * xwidth / 2)
                    r = radii[(x, y)]
                    modval = radiusmod.make_frame(t, 0, 0, duration - x / 20, duration)
                    if modval is None:
                        modval = 0                    
                    r += random.uniform(0, 3 * modval / 2)
          , xy=((x + 0.5) * xwidth, (y + 0.5) * ywidth), fill=(1, pinkfactor, pinkfactor, 0.15),
                                 stroke=(1, 0.3, 0.3), stroke_width=1).draw(surface)
                    alphafactor = random.uniform(0.2, 0.8)
                    endpoint = canim.make_frame(t, 0, 0, duration - y / 20, duration)
                    endpoint2 = canim2.make_frame(t, 0, 0, duration - x / 20, duration)
                    if endpoint is not None:
                        gizeh.arc(xwidth / 2, 0, endpoint, xy=((x + 0.5) * xwidth, (y + 0.5) * ywidth), fill=None,
                                  stroke=(1, 1, 1, alphafactor), stroke_width=2).draw(surface)
                    if endpoint2 is not None:
                        gizeh.arc(xwidth / 4, 0, endpoint2, xy=((x + 0.5) * xwidth, (y + 0.5) * ywidth),
                                  fill=(0, 1, 1, alphafactor),
                                  stroke=(1, 1, 1, alphafactor), stroke_width=2).draw(surface)
        return surface.get_npimage()

    clip = mpy.VideoClip(make_frame, duration=duration)
    clip.write_gif("example_heart1.gif", fps=fps, opt="nq")

Where can I find this magic vector tween library?

Glad you're asking! All the magic can be found here.

I'm curious to see what advanced animations you can come up with using a mask driven vector animation technique. Note: to add more layers of omgwow!!! nothing stops you from animating the mask image as well, or from generating the mask images programmatically. Or maybe you can go recursive.... how about you use the heart animation frames as animation masks for yet another, perhaps even more psychedelic animation?

You can also watch and listen to my recent music video "The Vows" that embeds vector animations generated using pyvectortween here:

Monday, May 1, 2017

Tweened vector animations in python part II


This is a continuation of part I on creating vector animations with python. In recent days, the library has seen the addition of many new features: 
  • animations along paths defined by symbolic parametric equations
  • animations along Bezier curves defined by control points (any order)
  • animations defined by equations in polar coordinates
  • composing animations into sequential and parallel animations
  • all animations now support getting a list of curve points for displaying the paths
And all of the new animations also still support tweening of course!

Show me! Show me!

The following picture shows a sequential animation of a cubic and a quadratic bezier segment, displaying the path, and using an easeOutBounce tweening on both segments.

The above picture is generated by one of the included examples in the vectortween library.

Sunday, April 23, 2017

Tweened vector animations with python


In this blog post I'll show how to create tweened vector animations with python. People who arrive here probably ought to look at Zulko's excellent blog post first to find the excellent gizeh library, written on top of cairo, that allows for generating vector graphics animations in python. And then there's also Zulko's wonderful moviepy library that allows saving these animations to animated .gif or one of the many video formats supported by ffmpeg.

While experimenting with Zulko's libraries, I was struck by what seemed a missing feature: a convenient way to introduce "easing" in animations. I subsequently found the pytweening library which implements the formulas required to perform easing and created my own vectortween library to wrap pytweening and make it usable in combination with Zulko's libraries (as well as other libraries).


Here's some sample code using plain gizeh. It creates a yellow circle moving from the top left to the middle of the drawing:

if __name__ == "__main__":
    import gizeh
    import moviepy.editor as mpy

    W, H = 250, 250  # width, height, in pixels
    duration = 5 # duration of the clip, in seconds
    fps = 25

    def make_frame(t):
       # prepare a drawing surface
       surface = gizeh.Surface(W, H)

       # as time t evolves from 0 to 5 seconds, calculate new position of
       # the yellow circle, xy=((t*22),(t*22)), fill=(1,1,0)).draw(surface)
       return surface.get_npimage()

    clip = mpy.VideoClip(make_frame, duration=duration)
    clip.write_gif("example0-plain.gif", fps=fps, fuzz=10)

And what does it look like?

Boring, right?

Now how about you wanted add some excitement by letting the yellow circle bounce a bit upon arrival in the middle of the picture? Are you ready to write out the math formulas to do so? If yes, by all means go ahead! If no, you could try the vectortweening library to write this:

if __name__ == "__main__":
    import gizeh
    import moviepy.editor as mpy

    from vectortween.PointAnimation import PointAnimation

    W, H = 250, 250  # width, height, in pixels
    duration = 5 # duration of the clip, in seconds
    fps = 25

    def make_frame(t):
        # prepare a drawing surface
        surface = gizeh.Surface(W, H)

        # p animates from position (0,0) to position (110,110) with an
        # easeOutElastic tweening method
        p = PointAnimation((0, 0), (110, 110), tween=['easeOutElastic', 1, 0.2])

        # circle to appear at second 0.2, animate from second 1 to 4,
        # then stay visible until disappearance in second 5
        xy = p.make_frame(t, 0.2, 1, 4, 5)

        # because graphics can disappear, we must check for None
        if None not in xy:
  , xy=xy, fill=(1,1,0)).draw(surface)
            return surface.get_npimage()

    clip = mpy.VideoClip(make_frame, duration=duration)
    clip.write_gif("example0-tween.gif", fps=fps, fuzz=10)

And what does it look like?

We want more, we want more...

Ok, ok... I hear you. Here's another one:

Ooooh I need this now!

Well of course you do... Here's all you need!

Saturday, March 12, 2016

Virtualbox on arch linux kernel 4.4.5

Since my last upgrade to latest packages of arch linux including kernel 4.4.5 I had trouble starting virtualbox.

[user@xxxxx ~]$ sudo /sbin/rcvboxdrv setup           Unloading modules:  Loading modules: modprobe: ERROR: could not insert 'vboxnetadp': Exec format error
modprobe: ERROR: could not insert 'vboxnetflt': Exec format error
modprobe: ERROR: could not insert 'vboxpci': Exec format error
modprobe: ERROR: could not insert 'vboxdrv': Exec format error
Here are the steps I took to make the trouble go away:
  • reboot to make the new kernel 4.4.5 active
    • sudo shutdown -r now
  • reinstall linux-headers (use the correct version for your kernel!): this triggered a recompile of the virtualbox kernel modules by dkms
    • sudo pacman -S linux-headers
  • remove package virtualbox-host-modules: this removed old pre-compiled vbox kernel modules. In retrospect, it was probably this last step that actually solved the problem.
    • sudo pacman -R virtualbox-host-modules
For what it's worth :)

Friday, August 21, 2015

Microsoft Random Limitation Rant

Welcome in 2015, the era of Multi-GHz Clocks, Multi-GB RAM, PetaByte storage and... 260 character path names.

Once again I was bitten by yet a random path length limitation on a win 7 64-bit platform with visual studio 2013. Of course I am aware that longer path lengths are supported on current windows platforms, provided you do some "\\?\" hocus pocus but Microsoft basically admits it's too much magic for them, given that they won't fix the random limitations embedded in their own build tools and just prefer to let hundreds (probably thousands) of developers bump into them.

Don't believe me? Read the giant's own website for more details and weep :)

More info about using long path names:

Monday, August 11, 2014

Why did flash player stop working in chromium on Debian?

Adobe flash is evil right? Yet many sites insist on using it (who knows for what evil purpose?!) and it's annoying the heck out of me if they keep throwing warnings about a missing flash player. 

Well, here's a clue: somehow I missed the news that chrome/chromium now uses a google maintained version of flash player which you can install using 
apt-get install pepperflashplugin-nonfree
Nothing to see here really. Carry on :)