Artium Nihamkin's Blog

Boeing 787 Dreamliner and the Counter Overflow

2020-02-29T23:00:00+02:00

This is another post in a series which covers famous software failures. Today's failure is not very old nor it is very famous.

In 2015, the Federal Aviation Administration (FAA) issued and Airworthiness Derivative (AD) regarding Boeing's flagship aircraft at that time, the 787 Dreamliner.

In the AD, they write:

We have been advised by Boeing of an issue identified during laboratory testing. The software counter internal to the generator control units(GCUs) will overflow after 248 days of continuous power, causing that GCU to go into failsafe mode. If the four main GCUs (associated with the engine mounted generators) were powered up at the same time, after 248 days of continuous power, all four GCUs will go into fail safe mode at the same time, resulting in a loss of all AC electrical power regardless of flight phase.

So what is going on here? Let's break down and analyze.

Discovery

First of all, this bug was discovered during lab testing. It was discovered after the aircraft became operational. How is it possible? There are two possible explanations that I can think of. First, it could have been discovered during the testing of the next software version. Other explanation is that FAA certifies aircraft even though the testing of the aircraft's components is not complete at the time of the certification. Powering the aircraft's avionics systems for such long period and checking that it still works properly can be such test.

Ok, so what is a GCU?

It stands for Generator Control Unit, this is an embedded device which controls the generation of electric power produced by rotation of the engines. Usually there is one generator for each engine but the Dreamliner has two.

Loosing the GCUs can potentially lead to losing all power. On modern aircraft, the pilot is not controlling the flight surfaces directly. Instead, stick and throttle movements, along with a bunch of sensor data, are used as an input into a flight computer. The computer calculates how the flight surfaces should be moved and outputs commands to actuators. Without power, this process can not work which means the pilot can not control the aircraft anymore.

Overflow

We also know that the GCUs went into a failsafe mode due to counter overflow after 248 days. So what is so special about 248 days?

248 days	248 X 24 X 3600 seconds	2,142,720,000 ✕10 milliseconds
2³¹		2,147,483,648
249 days	249 X 24 X 3600 seconds	2,151,360,000 ✕10 milliseconds

Based on this, we can guess that this counter is 32 bit signed integer and it is being incremented once every 10ms. Somewhere during the 248th day it overflows. If all the GCUs on the aircraft start approximately at the same time, they will go into failsafe mode at the same time and thus could leave the aircraft without power. This is not fatal as there is a backup generator powered by the flow of air which deploys in such situations. Nevertheless this is a very dangerous situation.

Solution

Until a software fix was ready, the AD requires aircraft operators to cycle power to the GCUs every 120 days.

AD Requirements

This AD requires a repetitive maintenance task for electrical power deactivation.

Interim Action

We consider this AD interim action. The manufacturer is currently developing a GCU software upgrade that will address the unsafe condition identified in this AD. Once this software is developed, approved, and available, we might consider additional rulemaking.

I don't know how common it is for an aircraft to operate for 120 days continuously powering it's electrical systems. The 787 is not an aircraft that you power off "at the end of the day". There is no such concept at all as every minute the aircraft is not in the air, the operator is losing money. It might even go to maintenance while still being powered on.

Airbus A350

In 2017, on the other side of the ocean, Airbus had a similar issue. This time it was an overflow every 149 hours.

For the sake of completeness hare is a link to the AD published by European Union Aviation Safety Agency (EASA) and a similar table as for the 787 case:

149 hours	149 X 3600 seconds	536,400,000 ✕ milliseconds
2²⁹		536,870,912
150 hours	150 X 3600 seconds	540,000,000 ✕ milliseconds

My guess is that the counter is a signed fixed point number with an LSB of 0.25, but this is of course just a speculation.

Zune Leap Year Freeze

2020-01-25T02:00:00+02:00

I am starting a new series of blog posts in which I will do analysis of famous bugs and system design failures related to software. I will attempt to present more technical analysis than what is available when reading about these stories in the mainstream media.

I love reading about "failure" stories. It is a great entertainment but can also be an educating experience which inspire me to think or do things differently. Hope it will be relevant to you as well.

Zune

Zune 30 was a 30GB media players developed by Microsoft and released in 2006. It was the era of the iPod and Microsoft wanted a portable media player as well.

At midnight of December 31, 2008, all the millions of Zune 30 that Microsoft sold started malfunctioning. When turned on, they would freeze on the loading screen.

Before looking at the code that caused the bug, lets understand how consumer electronic products are usually developed.

A hardware component manufacturer develops a component tailored for specific product class. For example a DSP chip for surveillance cameras, a WIFI communication chip for cell phones or a system on chip with all the necessary peripherals for a portable media player.

Along with this chip, the manufacturer develops a reference product. The design of the reference product is released and includes design documents, schematics, PCB layout and relevant code.

A consumer product manufacturer will not do R&D from scratch, it will license the reference design and "differentiate" it. Each manufacturer changes the design slightly and thus we get a range of different product.

In our case microsoft used the Freescale MC13783 chip. This chip handles power management and audio and is designed specifically to be used in portable media players.

Among other things, this chip handles the real time clock. This is a clock powered by an auxiliary power source and is supposed to keep time when the device is turned off.

At startup, Zune's main processor would load the time from the real time clock in days since 1980 and convert to a proper date (day, month, year). Here is a snippet of a driver code that does the conversion to the current year:

BOOL ConvertDays(UINT32 days, SYSTEMTIME* lpTime)
{
    ...
    year = ORIGINYEAR; /* = 1980 */

    while (days > 365)
    {
        if (IsLeapYear(year))
        {
            if (days > 366)
            {
                days -= 366;
                year += 1;
            }
        }
        else
        {
            days -= 365;
            year += 1;
        }
    }
    ...
}

Can you spot the bug?

Before revealing the answer, let me help with some clues:

Zune froze, how can this snippet cause the device to freeze?

.
.
.
.
.
.
.
.

What is special about December 31?

.
.
.
.
.
.
.
.

What is special about 2008?

.
.
.
.
.
.
.
.

Let's address these three questions one by one. The freezing could have been caused by an infinite loop. This happens if days never gets smaller than 366.

The date of this incident is December 31, which is the last day of the year. It could be the 365th day of that year or if it is leap year it will be the 366th day. Speaking of a leap year, 2008 was indeed a leap year.

After several iterations of the while loop, we end up with days equal 366. The code enters the first if but does not enter the second. The state during the execution of the next iteration is identical since nothing has changed, and thus the loop will run infinitely.

Full disclosure, here is a link for reference I used for this blog post.

Obviously this issue resolved itself on January the first. Until next leap year at least, but I doubt there were many Zune users at that time.

Takeaway

1. Non simple recalculation of the loop counter inside the body of the loop is a smell. I have seen coding standards that forbid the use of while loops and force the for loops to be "well formed", partly because of this reason.

2. Time and date calculations are complicated and have a lot of edge cases. You can watch this video to get a reinforcement of this view.

Playing WAV Files on ESP8266

2019-09-15T03:00:00+03:00

In the previous post, ESP8266 was playing music based on pitch sequences and a recording of a sine wave. This is not sufficient for the speech sythesis system. A method for playing WAV files is required.

Fortunately, WAV format uses PCM to represents the waveform. Thus the data can be directly fed into PCM5102.

For a test, I synthesized a simple WAV file using desktop version of Festival.

Your browser does not support the audio element.

Next, I parsed this file using Python script. This description of the WAV format was very helpful.

The script can be found at this link. It's input is a WAV file and it produces an H file looks like this:

#ifndef WAV_DATA
#define WAV_DATA

#define NUM_SAMPLES 33442
#define SAMPLE_RATE 16000

int16_t data[NUM_SAMPLES] PROGMEM = {

    .... // samples

};
#endif

Please keep in mind that I work with WAV files produced by Festival, and thus the script assumes that the input WAV file has 1 channel and 16 bits per sample. It can be extended to support any kind of WAV file. For me this was not worth the effort since the script is used only for testing/experimenting.

This H file is included into the most simple Arduino sketch:

#include "ESP8266WiFi.h"
#include "i2s.h"
#include "i2s_reg.h"

#include "wav_data.h"

void setup() {
    system_update_cpu_freq(160);
    i2s_begin();
    i2s_set_rate(SAMPLE_RATE);
}

void loop()
{
    for(size_t i = 0; i < NUM_SAMPLES; i++) {
        uint16_t ua = pgm_read_word_unaligned( &(data[i]));
        int16_t a;
        memcpy(&a, &ua, sizeof(int16_t)); // reinterpret bits of unsigend into a signed value
        i2s_write_lr(a,a);
    }

    delay(3000);
}

It looks straight forward except the use of PROGMEM macro and the strange memory access.

The WAV file is too big to fit into ESP8266 RAM, therefore it has to be stored on the FLASH and the samples have to be read from there. PROGMEM macro hides a compiler directive that tells it to put the data array into the irom.text section.

Although most of the ESP8266 FLASH is memory mapped, only 32 bit aligned access is allowed. Since the samples in the array are 16 bit of length, it is not possible to access individual array element even if the whole array is correctly aligned.

The pgm_read_word_unaligned function is a built in way of accessing a 16 bit word from FLASH. Behind the scenes it calculate the aligned address, retrieves a 32 bit value and returns only the relevant part of it in the form of 16 bit unsigned value. The memecpy call is just a portable way to reinterpret the unsigned value into a signed value (that is what stored in the data array in the first place).

Since this is part of a project with a goal to run speech synthesis on the ESP hardware, the next thing is to actually port flite to ESP.

PCM Audio on ESP8266 using the PCM5102 Chip

2019-08-22T23:00:00+03:00

One of the projects I am currently working on is a text to speech system based on ESP8266 (or more realistically - ESP32). It will utilize the CMU Flite speech synthesis library. But a way to play back the synthesized waveform is also required.

For this propose, I will use a PCM5102 chip that will convert digital data sent from the ESP8266 over I2S to analog signal that drive the connected speaker.

ESP2866 has a hardware implemented I2S, so specific pins are required. Here is the table showing the connections I have made.

ESP8266 Name	ESP8266 IO	PCM5102 Pin
MTDO	IO 15	BC(L)K
U0RXD	IO 3	DIN
GPIO2	IO 2	L(R)CK
Connect to 3.3V		VIN
Connect to GND		GND

If you want to use the other pins which control certain aspects of the chips behavior, you can refer to the data-sheet. I did not connect any other pin on the breakout board of the PCM5102.

The full I2S protocol is explained on wikipedia better than anything I could write. Here are few bullets to help you understand the code which appear later:

Data words are 16 bits two's complement.
Each data word represents the magnitude of the wave front at specific point in time (this is called PCM)
There are two data words for each sample representing left and right channels.
The sample rate is, as the name suggests, number of samples (and not words) per second.
Here, I will not utilise stereo, so each sample's left and right channels will be identical.

Ok, so first let's create 512 samples of a sine array using a python script:

import matplotlib.pylab as plt
import numpy as np

def chunks(l, n):
    """Yield successive n-sized chunks from l."""
    for i in range(0, len(l), n):
        yield l[i:i + n]

SAMPLES = 512

x = np.linspace( 0, 2.0*np.pi, SAMPLES)
y = list( (int(a) if a >= 0 else int(2**16 + a) for a in  (np.sin(x)) * 2**15 ) )

for row in chunks(y, 8):
    print ("".join(("{0:#0{1}x}, ".format(word,6) for word in row)))

plt.plot(x, y)
plt.show()

The above script prints the samples in a format suitable for embedding into a C source file.

Few things to notice:

The graph does not look like a sinus. This is because we are looking at unsigned values which represent signed data. So for example 0x8000 is actually -32768.
The start of the wave is at the value of 0, which represents the diaphragm of the speaker at resting position. Other values would mean a sudden jump to the requested position, which will add a little bit of noise.
There is no no intention to use stereo. The same values will be sent to both channels. Thus the "samples" are single data words and not pairs.

Using this data it will be possible to generate sine waves of various frequencies (also known as pitches).

The ESP8266 implementation of Arduino environment has a built in library for sending I2S data. All the function signatures can be found in i2s.h.

In the setup function, I set up the I2S to use 44100 Hz sample rate and the CPU to run at 160 MHz:

#include "ESP8266WiFi.h"
#include "i2s.h"
#include "i2s_reg.h"


#define RATE 44100

void setup() {
system_update_cpu_freq(160);

i2s_begin();
i2s_set_rate(RATE);
}

On the ESP8266, I created a function that is similar to Arduino's tone function. But this one is not a puny, square wave, piezzo buzzer driver:

#define SINE_SAMPLES 512

int16_t sine[SINE_SAMPLES] = { ... } // Paste here the generated samples from the python script

/**
* Outputs a sine waveover I2S of the requested frequency for the
* requested duration of milliseconds.
*/
void writeNote(uint32 freq, uint32 duration)
{
    if (freq == 0 || duration == 0) {
        return;
    }
    // Number of samples required for the requested duration
    uint32 totalSamples = (RATE * duration) / 1000;

    // Number of samples in full period of a sine wave for that particular frequency
    uint32 samplesPerPeriod = RATE / freq;

    // Round down total samples to an integer number of sine periods
    totalSamples = (totalSamples / samplesPerPeriod) * samplesPerPeriod;

    for (uint32 i = 0; i < totalSamples; i++) {
        uint32 sampleIdx = map(i % samplesPerPeriod, 0, samplesPerPeriod, 0, SINE_SAMPLES);
        i2s_write_lr( sine[sampleIdx],  sine[sampleIdx]);
    }

    delay(50);
}

You can hear in the following video, the tune of star wars played using writeNote function. The full source of the Arduino sketch can be found here.

Next thing I will do is playing back WAV files. Stay tuned ;)

Better TFT Library for ESP8266

2019-08-20T03:00:00+03:00

In 2016 I wrote a tutorial about connecting ESP8266 with an ili9341 TFT display. That tutorial suggested using Adafruit's library modified to work with the ESP8266.

Now I discovered that there is a much better library which is tailored for the ESP8266. This library is called TFT_eSPI.

This library is available from Arduino's library manager, thus the process of installation is very easy.

One thing to notice is that after installation, it is required to go to the file User_Setup.h which is in the librarie's directory and edit it. There are a lot of parameters there, the important are the definition of the ESP8266 pins that will be used for connecting the TFT display:

For example, for my Lolin NodeMCU V3 I used the following configuration:

#define TFT_CS   PIN_D2  // Chip select control pin D8
#define TFT_DC   PIN_D1  // Data Command control pin
//#define TFT_RST  PIN_D4  // Reset pin (could connect to NodeMCU RST, see next line)
#define TFT_RST  -1

Also check that ILI9341 driver is selected (TFT_eSPI supports several screen models):

// Only define one driver, the other ones must be commented out
#define ILI9341_DRIVER
//#define ST7735_DRIVER      // Define additional parameters below for this display
//#define ILI9163_DRIVER     // Define additional parameters below for this display
//#define S6D02A1_DRIVER
//#define RPI_ILI9486_DRIVER // 20MHz maximum SPI
//#define HX8357D_DRIVER
//#define ILI9481_DRIVER
//#define ILI9486_DRIVER
//#define ILI9488_DRIVER     // WARNING: Do not connect ILI9488 display SDO to MISO if other devices share the SPI bus (TFT SDO does NOT tristate when CS is high)
//#define ST7789_DRIVER      // Full configuration option, define additional parameters below for this display
//#define ST7789_2_DRIVER    // Minimal configuration option, define additional parameters below for this display
//#define R61581_DRIVER
//#define RM68140_DRIVER

The API is similar to Adafruit in most cases, but additionally TFT_eSPI has more features like ability to draw Sprites.

As a demonstration of how quicker it is, here are few comparisons of the graphics test program. I tried several SPI frequencies, this can be controlled using the #define SPI_FREQUENCY macro in User_Setup.h.

Adafruit_ILI9341 Library

ILI9341 Test!
Display Power Mode: 0x94
MADCTL Mode: 0x48
Pixel Format: 0x5
Image Format: 0x80
Self Diagnostic: 0xC0
Benchmark                Time (microseconds)
Screen fill              1606040
Text                     92026
Lines                    809972
Horiz/Vert Lines         134436
Rectangles (outline)     85586
Rectangles (filled)      3329421
Circles (filled)         421733
Circles (outline)        356141
Triangles (outline)      184657
Triangles (filled)       1121997
Rounded rects (outline)  166946
Rounded rects (filled)   3330265
Done!

TFT_eSPI with SPI_FREQUENCY set to 27000000

TFT_eSPI library test!
Benchmark                Time (microseconds)
Screen fill              237671
Text                     27003
Lines                    160091
Horiz/Vert Lines         20537
Rectangles (outline)     13842
Rectangles (filled)      487678
Circles (filled)         115760
Circles (outline)        101732
Triangles (outline)      37935
Triangles (filled)       194903
Rounded rects (outline)  48270
Rounded rects (filled)   554112
Done!

TFT_eSPI with SPI_FREQUENCY set to 40000000

TFT_eSPI library test!
Benchmark                Time (microseconds)
Screen fill              161174
Text                     23332
Lines                    137703
Horiz/Vert Lines         14246
Rectangles (outline)     9835
Rectangles (filled)      330783
Circles (filled)         93076
Circles (outline)        87518
Triangles (outline)      32453
Triangles (filled)       142416
Rounded rects (outline)  39809
Rounded rects (filled)   380746
Done!

TFT_eSPI with SPI_FREQUENCY set to 80000000

TFT_eSPI library test!
Benchmark                Time (microseconds)
Screen fill              84359
Text                     20697
Lines                    118085
Horiz/Vert Lines         7990
Rectangles (outline)     5913
Rectangles (filled)      173302
Circles (filled)         71464
Circles (outline)        78870
Triangles (outline)      27596
Triangles (filled)       91296
Rounded rects (outline)  33468
Rounded rects (filled)   207793
Done!

TFT_eSPI with SPI_FREQUENCY set to 80000000 and CPU running at 160 MHz

TFT_eSPI library test!
Benchmark                Time (microseconds)
Screen fill              81344
Text                     13412
Lines                    79826
Horiz/Vert Lines         7326
Rectangles (outline)     5143
Rectangles (filled)      166979
Circles (filled)         51367
Circles (outline)        51259
Triangles (outline)      18983
Triangles (filled)       74893
Rounded rects (outline)  22796
Rounded rects (filled)   194053
Done!

Building Text to Speech Service Based on Festival

2019-06-14T13:00:00+03:00

Your browser does not support the audio element.

Last week I participated in Hackathon where we built a demo of domain specific personal assistant. One of the components of such system is the text to speech component.

In this post I will explain how I installed Festival speech synthesis system, how I managed to alter the system to fit our specific domain requirements and how I wrapped it with a Node.js server.

Festival

Festival is a speech synthesis system (also called speech to text, STT or S2T). It has two maintainers and therefore two websites (which might be confusing a bit):

Another thing one should know about festival is the difference between Festival and festvox. Festival is the speech synthesis system itself. On the other hand, festvox is the name of a set of tools used to create voices for Festival. Sometimes the whole project (Festival + festvox) are called by the festvox name, so do not be confused.

Basic Installation on Linux

Although Festival can be installed on some distributions from a package manager like apt, we will build it from source.

I tested these instructions on Ubuntu 18.04 and 16.04.

At the time of writing this blog post, the latest version of Festival is 2.5.

Got to http://www.festvox.org/packed/festival/2.5/ and download all five tar.gz archives that appear there.

Additionally, got to the voices subdirectory and download:

festvox_kallpc16k.tar.gz
festvox_rablpc16k.tar.gz
festvox_cmu_us_awb_cg.tar.gz
festvox_cmu_us_awb_cg.tar.gz

You can download additional voices, but these are basic.

Put all the files in a single directory.

Now extract all the archive using the following command:

for a in `\ls -1 *.tar.gz`; do gzip -dc $a | tar xf -; done

The directory structure will now look like this:

~/Festival$ ls -l
total 126804
drwxr-xr-x  9 artium artium     4096 May 26  2018 festival
-rw-rw-r--  1 artium artium   789013 Jun  8 13:49 festival-2.5.0-release.tar.gz
-rw-rw-r--  1 artium artium  1925748 Jun  8 13:49 festlex_CMU.tar.gz
-rw-rw-r--  1 artium artium  1474233 Jun  8 13:49 festlex_OALD.tar.gz
-rw-rw-r--  1 artium artium   243656 Jun  8 13:49 festlex_POSLEX.tar.gz
-rw-rw-r--  1 artium artium 52653464 Jun  8 23:56 festvox_cmu_us_awb_cg.tar.gz
-rw-rw-r--  1 artium artium 61933982 Jun  8 23:56 festvox_cmu_us_rms_cg.tar.gz
-rw-rw-r--  1 artium artium  4103663 Jun  8 13:49 festvox_kallpc16k.tar.gz
-rw-rw-r--  1 artium artium  5369131 Jun  8 13:49 festvox_rablpc16k.tar.gz
drwxr-xr-x 23 artium artium     4096 Jun  8 23:50 speech_tools
-rw-rw-r--  1 artium artium  1328624 Jun  8 13:49 speech_tools-2.5.0-release.tar.gz

Notice that only two new directories were created. This is because all archives share the same directory structure and thus when extracting files, they go into the correct place inside festival automatically.

Now make sure ncurses5 is installed:

sudo apt install libncurses5

Now go inside the speech_tools directory and run:

~/Festival/speech_tools$ ./configure

Followed by:

~/Festival/speech_tools$ make

Just for reference, here is the compiler setup I used:

$ gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/7/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 7.4.0-1ubuntu1~18.04' --with-bugurl=file:///usr/share/doc/gcc-7/README.Bugs --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++ --prefix=/usr --with-gcc-major-version-only --program-suffix=-7 --program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-libmpx --enable-plugin --enable-default-pie --with-system-zlib --with-target-system-zlib --enable-objc-gc=auto --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-offload-targets=nvptx-none --without-cuda-driver --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 7.4.0 (Ubuntu 7.4.0-1ubuntu1~18.04)

After building speech_tools it is possible to build Festival itself. There is no need to put speech_tools/bin in the PATH for this purpose. Just make sure you follow the directory hierarchy I showed earlier.

Run:

~/Festival/festival$ ./configure

Followed by:

~/Festival/festival$ make

After building is finished, if you run make test some tests will fail. Some because the copyright message of the latests version was changed, some because of missing voices and others because reasons I do not understand. For reference, here is the output I got.

These failing tests are not a no-go situation. We can use festival with no problem. But first lets add it into PATH (change the path of bin to reflect your location):

export PATH=$PATH:/home/artium/Festival/festival/bin

Now we can run festival which will start the interpreter.

Let's test it. Notice that to exit the interpreter you need to press Ctrl+ D.

festival> (SayText "hello")
#<Utterance 0x7f25e1a58690>
festival> Linux: can't open /dev/dsp

You probably does not hear anything and get this can't open /dev/dsp message. The reason is that Ubuntu does not have the OSS modules by default. There are two ways to solve this:

Install osspd (sudpo apt-get install osspd)
Tell festival to use ALSA for sound output.

The second solution is highly recommended. You need to open ~/.festivalrc file and add these lines:

(Parameter.set 'Audio_Command "aplay -q -c 1 -t raw -f s16 -r $SR $FILE")
(Parameter.set 'Audio_Method 'Audio_Command)

Now running the SayText command should output the speech through the speakers or headphones. Make sure you have output device attached to the computer.

Saving to File

To save a WAV file, many online tutorials recommend the text2wave program that was build along with festival. For me, running it gives a segfault, so here is an alternative way using festival itself.

(utt.save.wave (SayText "This is an example text") "test.wav" 'riff)

This method is ok for testing but the problem is that SayText will actually say the text and only then the file will be saved.

Here is a better way to accomplish the same task:

(set! utt1 (Utterance Text "This is an example text"))
(utt.synth utt1)
(utt.save.wave utt1 "test2.wav" 'riff)

Modifying the System

One of the requirements of our domain specific personal assistant is that numbers should be pronounces digit by digit. For example the number "300.25" should be pronounced "three zero zero two five" and not "three hundred dot twenty five"

Using SABLE markup it is possible to use the <SAYAS MODE="literal"> tag. But I wanted this to work with text mode not sable mode. I did not find a built-in way to tell festival to use literal pronunciation when synthesizing an utterance. So I had to implement it myself.

I followed the tutorial here and used the example on the bottom of chapter 5 to figure out what change should I make to festival.

The result is the following code that should be executed after setting a voice. I will not get into details of festival's architecture and Scheme syntax here.

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;;                                                                 ;;
;; Custom definition required for our personal assistant project   ;;
;; Required because domain specific requirement.                   ;;
;;                                                                 ;;
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

(set! previous_token_to_words token_to_words)

(define (token_to_words token name)
    "In case of a real number, will pronounce the digits one by one excluding the decimal point"
    (cond
        (
            (string-matches name "[0-9]+\\.[0-9]+" )
            (mapcar num-to-literal (filter is_digit (symbolexplode name))) ; ;; (utf8explode utf8string) ?
        )
        (
            t
            (previous_token_to_words token name)
        )
    )
)

(define (num-to-literal num)
    "Returns the literal word of numeric characters"
    (car
        (cdr
            (assoc
                (parse-number num)
                '(
                    (1 one) (2 two) (3 three)
                    (4 four) (5 five) (6 six)
                    (7 seven) (8 eight) (9 nine)
                    (0 zero)
                )
            )
        )
    )
)

(define (filter pred lst)
    "A new list is returned containing only the item matching the predicate"
    (reverse (filter-help pred lst '()))
)

(define (filter-help pred lst res)
    (cond
        (
            (null? lst) res
        )
        (
            (pred (car lst))
            (filter-help pred (cdr lst)  (cons (car lst) res))
        )
        (
            t
            (filter-help pred (cdr lst)  res)
        )
    )
)


(define (is_digit input)
    "Is input a single digit number?"
    (string-matches input "[0-9]")
)

Here is an example of synthesis of the phrase "The radio frequency is 432.78":

Your browser does not support the audio element.

Web Service

For the web service I used Node.js with Express.

My strategy was:

Receive the a text through a POST request.
Generate a scheme code that would synthesize this text into speech and save the speech into WAV file.
Save the scheme code in a temporary script file.
Invoke a festival process in batch mode and give it the scheme script as a parameter.
When finished synthesis, use Express's res.download API call to send a response with the WAV file.

Here is an example of festival.js helper module:

const path = require("path")

const prod_dir = "/tmp/"

exports.get_sys_err_path = function (cb) {
    cb(prod_dir + "static/error_in_tts.wav")
}

exports.get_sound_file_path = function (text, cb) {

    var timestamp = Date.now().toString();

    var fs = require("fs");

    var voice = "cmu_us_fem_cg"

    // Important: must call vocie BEFORE custom require
    var data = `(voice_${voice})`
    // Notice: path delimiter might not be good for Windows
    data += `(require "${path.resolve(".")}/custom")`
    data += `(set! utt1 (Utterance Text "${text}"))`
    data += '(utt.synth utt1)'
    data += `(utt.save.wave utt1 "${timestamp}.wav" 'riff)`

    var scriptFullPath = prod_dir + timestamp + '.scm'
    fs.writeFile(scriptFullPath, data, (err) => {
        if (err) console.log(err);
        console.log("Created file: " + scriptFullPath);
    });

    // Precondition: festival is in $PATH
    const
        { spawn } = require('child_process'),
        ls = spawn('festival', ['--batch', timestamp + '.scm'], { cwd: prod_dir });

    ls.stdout.on('data', data => {
        console.log(`stdout: ${data}`);
    });

    ls.stderr.on('data', data => {
        console.log(`stderr: ${data}`);
    });

    ls.on('close', code => {
        console.log(`child process exited with code ${code}`);

        if (code != 0) {
            Festival.get_sys_err_path(function(filepath) {
                cb(filepath)
            })
        } else {
            cb(prod_dir + timestamp + '.wav')
        }

    });
}

The complete code of the service can be found in this repository.

This process is not very efficient (and the implementation is not secure. Can you spot why?)

I measured the time to receive a response from the server for the most simplest text: I. it was 1.11 seconds.

Here are some examples of the average response times of different phrases I measured:

Phrase	Time
I	1.1s
Test	1.2s
This is a test	1.4s
This is a longer test	1.8s
The frequency was set to 30000	2.4s

And that is just the "text to speech" part. When integrated with all the other steps of the personal assistant pipeline, we got a response latency that was hardly acceptable. A user should "feel" the interaction with the assistant thus the program can not "stop" and think about a reply every time a simple question is asked. It was good enough for a Hackathon and a demonstration of a proof of concept, so we did not attempt to optimize.

Anyway, here are some ideas of why it takes that much time (no particular order):

There are several reads and writes of files for each request.
Each request causes a full Festival process to be forked. Initialization of the process and Festival itself takes time.
Network latency.
Downloading the WAV file (~100KB) takes time (bandwidth).
Festival itself is slow in synthesizing the WAV files.

We could use festival's built-in server capability to overcome the disk writes and process forking issues.

As for latency, bandwidth and general slowness of Festival, the best course of action in my opinion is to use Flite instead of Festival and do the synthesis locally on the client itself.

Filtering Meetup Organizers with Greasemonkey Script

2018-12-08T18:00:00+02:00

I love meetups. It is a great opportunity to learn new things and expose myself to new ideas. I manly interested in the technological stuff, but I am not picky. Great concepts, ideas and life lessons has the quality of being applicable across multiple fields.

Every two weeks or so, I go to meetup.com, choose "All Meeteups" and browse for interesting meetings in the following two weeks.

The problem is that there is a lot of clutter. Meetup allows to select which organizers I am interested in, but I want to select which organizers I am not interested in. I still want to see new stuff, but if I decided that something is not relevant to me, then I want to filter it out in the future.

Meetup does not allow filtering out specific organizers. So I decided to write a Greasemonkey script to do this for me.

Greasemonkey is plugin for firefox that allows running user defined scripts written with Javascript. The scripts will run only when a page matching a specific pattern is opened by the browser.

Here is the script which does the filtering. It is pretty simple. You can also install it from here.

// ==UserScript==
// @name     Filter Out Meetup Search Results
// @version  4
// @description Remove items from search results of meetup.com based on regular expression. Personally I use this to filter out organizers that are not relevant to me.
// @author      Artium Nihamkin artium@nihamkin.com
// @icon https://secure.meetupstatic.com/s/img/68780390453345256452178/favicon.ico
// @homepageURL    http://www.nihamkin.com
// @include https://www.meetup.com/*
// @require http://ajax.googleapis.com/ajax/libs/jquery/1.9.1/jquery.min.js
// ==/UserScript==

function f() {

    // You need to edit this with your own filtering keywords. The regular expression
    // is run against href attributes of the search result links.
    // See http://www.nihamkin.com/filtering-meetup-organizers-with-greasemonkey-script.html
    //
    var blacklist = ["All-the-best-Workshops-DIY-Israel", "The-Best-Science-Bar-Talks-in-Tel-aviv"];
    var blRegex = new RegExp(blacklist.join("|"));

    var links = $("a").filter(function(i,el) { return (blRegex.test(el.href)); })
      .closest("li.event-listing")
      .remove();
}

document.body.addEventListener("DOMNodeInserted", f, true);

First, some comparisons before and after:

Now I will explain how function f works. The blacklist array is an array of patterns. These patterns should match the url of the meetup pages you want to filter out. Since all meetup page urls contain the organizer name, I used it as a filter criteria.

After building the regular expression, the filtering will happen. This task is assisted by jquery library which is imported in the @require line.

First it finds all a tags whose href match the regular expression, jquery is asked to look up the DOM tree for the first occurrence of li tag with event-listing class. This li tag is the container of the whole box, and thus it is removed to hide it.

Finally it is important to tell when function f should actually run. I hooked the DOMNodeInserted event because the meetup search results page is dynamic. It use Ajax to load more items as the user scrolls down and presses the "More" button. Therefore it is important to filter out any new search results as they appear on the page.

In conclusion, if you are familiar with Javascript then Greasemonkey can be a simple but powerful tool for personalizing the experience you have with your frequently visited websites.

How to Detect First Boot After Burning Program to Flash

2018-10-12T22:00:00+03:00

I will write about Arduino, but this technique is relevant for any embedded situation.

Many applications require retaining data such settings or parameters between resets. On the Arduino, the already existing EEPROM of the Atmega chip is a common solution for this problem.

For the Arduino, the EEPROM library can be used to write or read values from the EEPROM.

It is important that the program does not read from the EEPROM on the first time it is run after it was burned. The reason is that the version that was burned, might have different representation of the data than the one stored in the EEPROM. This will cause "not so nice" problems that might not go away even after resetting.

To solve this, one need to know when a program is run the first time after it was burned, and use default values instead of reading from the EEPROM (and of course store these to the EEPROM for next time)

This is sometimes called IPL (Initial Program Load) which means the first time program is loaded into the RAM.

To detect IPL one can use a constant that changes every time a new version of your program is produced. This constant is compared with a value stored in the EEPROM, and if they are different, it means that a new program was burned and this is it's first boot after burning. After this, the value is stored in the EEPROM and subsequent boots will not be detected as IPL.

This constant can be managed manually, can be some preprocessor macro representing git commit hash or date and time macro.

For the Arduino example below, I used date and time macros which I find simple and robust:

/**
 * Address of the version data. This should never change between versions.
 */
#define EEPROM_ADDR_VERSION_DATE 0x0

/**
 * Your data should start after this location.
 * On gcc, __DATE_ is 11 chars long, __TIME__ is 8
 */
#define EEPROM_ADD_START_OF_DATA 0x13

/**
* Determine if this is the first power-up after
* fresh program was loaded into the flash.
*/
boolean is_initial_program_load()
{
    const String version_date = __DATE__ __TIME__;
    uint16_t len = version_date.length();
    boolean is_ipl = false;

    for (unsigned int i = 0; i < len; i++) {
        int addr = EEPROM_ADDR_VERSION_DATE + i;

        if (EEPROM.read(addr) != version_date[i]) {
            EEPROM.write(addr, version_date[i]);
            is_ipl = true;
        }
    }

    return is_ipl;
}

Line Rasterisation

2018-08-06T21:00:00+03:00

Sometimes even the simplest things are not as simple as they look.

Recently, I had a task that required me to understand circle rasterisation. So I went back to my computer graphics course notes which I took many years ago to remind myself about how lines are drawn on a computer screen.

Lines are mathematical objects, on the other hand a screen is built out of a pixel grid of a finite resolution.

Even if the endpoints of the line correspond exactly to specific pixels, the line itself is not. It needs to be approximated by these pixels.

The process of deciding which pixels should be turned on to approximate the line is called rasterisation.

Reading the material again after that many years, I sharpened my understanding a bit and now hope to share some of the insights.

Naive algorithm

The simplest way to implement this is to step over every integer value of x and use y = mx + d function to calculate y. The value of y is not an integer, it is a real number. And since pixel grid is an integer grid, it needs to be rounded to the closest integer.

I will use processing tool for the demonstration:

Please notice that 300 is the height of the drawing area and the horizontal axis is inverses. Additionally, the plot_y offset parameter has nothing to do with the algorithm. It is just my way to control that each implementation plots it's result on a different on the screen. These is the reason for the 300 - (round(y) + plot_y_offset) calculation.

For simplicity, through this article, I will consider only lines with 0 < m < 1 and x0 < x1 (lines which are vectors in the first octant). Other cases (there are 8 of them) can be solved similarly by a combination of reversing the order of the points, by iterating over y instead of x and by reflecting around x axis. Also vertical, horizontal and diagonal lines have to be dealt with as edge cases.

/*
 * Draw a line pixel by pixel using the most naive algorithm
 */
void line_naive(int x0, int y0, int x1, int y1, int plot_y_offset)
{
  float m = float(y1 - y0) / float(x1 - x0);
  float b = y0 - m * x0;

  for (int x=x0; x<=x1; x++) {
    float y = (float(x) * m + b);
    set(x, 300 - (round(y) + plot_y_offset), black);
  }
}

There are several inefficiencies in this implementation.

For every pixel, there is a floating point multiplication.
For every pixel, there is a floating point rounding operation.
For every line, there are several more floating point operations, including multiplication and division.

Number 3 is relevant if you plan to draw large number of lines, which most of the applications require.

Digital Differential Analyzer

Digital Differential Analyzer (DDA) is just a fancy way of saying lets do increments instead of multiplication.

Naive implementation calculates float(x) * m every iteration of the loop. DDA on the other hand uses accumulator and increments by m every iteration. This replaces single floating point multiplication, with an addition.

/*
 * Draw a line pixel by pixel using the DDA technique:
 */
void line_dda(int x0, int y0, int x1, int y1, int plot_y_offset)
{
  float m = float(y1 - y0) / float(x1 - x0);
  float b = y0 - m * x0;
  float curr_y = y0;

  for (int x=x0; x<=x1; x++) {
    set(x, 300 - (round(curr_y) + plot_y_offset), black);
    curr_y += m;
  }
}

Bresenham's Midpoint

This step actually does not make the algorithm more efficient. It is an intermediate step before the final version of the algorithm.

Rounding should be not slower than comparing two floating point numbers because that is what actually rounding is about. More than that, depending on the implementation, rounding can be implemented efficiently in hardware and it could be faster.

/*
* Draw a line pixel by pixel using the Bresenham's midpoint algorithm
*/
void line_midpoint(int x0, int y0, int x1, int y1, int plot_y_offset)
{
  float m = float(y1 - y0) / float(x1 - x0);

  float midpoint_f = (float)y0 + 0.5;
  float curr_y = (float)y0;
  int plot_y = y0;

  set(x0, 300 - (y0 + plot_y_offset), black);
  for (int x = x0+1; x <= x1; x++) {

    if (curr_y > midpoint_f) {
      plot_y ++;
      midpoint_f += 1.0;
    }
    curr_y += m;

    set(x, 300 - (plot_y + plot_y_offset), black);
  }
}

Now, since you know the motivation behind using comparison, let's dive into the details.

If we look at a pixel (x,y) that was drawn at some iteration step of the loop, there are only two possible locations for the next pixel. Either one pixel to the right (x+1,y) or one pixel to the right and one pixel up (x+1,y+1). This behavior is due to constraints that were put on the slope and the order of the points.

So how to decide which pixel to take? One way already demonstrated was to calculate the real y and round it to the nearest integer value.

Another way is to calculate the middle point between these two pixels and to compare it to y. If the midpoint (sometimes called a threshold) is higher, then the algorithm draws the upper right pixel, otherwise it draws the pixel to the right.

The midpoint is updated incrementally every time pixel's y is changed.

Bresenham's Midpoint with Integer

Now comes the final trick. Can you implement the previous algorithm without using a single floating operation? Jack Elton Bresenham could do this.

/*
* Draw a line pixel by pixel using integer only operations in the for loop.
*/
void line_midpoint_integer(int x0, int y0, int x1, int y1, int plot_y_offset)
{
  int dx = x1 - x0;
  int dy = y1 - y0;

  int midpoint = (2*y0 + 1) * dx; // (y0 + 0.5) * 2.0) * dx;
  int midpoint_inc = 1 * 2 * dx;
  int curr_y = y0 * 2 * dx;
  int curr_y_inc = 2 * dy;        // m * 2.0 * dx

  int plot_y = y0;

  set(x0, 300 - (y0 + plot_y_offset), black);
  for (int x = x0+1; x <= x1; x++) {

    if (curr_y > midpoint) {
      plot_y ++;
      midpoint += midpoint_inc;
    }
    curr_y += curr_y_inc;

    set(x, 300 - (plot_y + plot_y_offset), black);
  }
}

Since the algorithm draws on pixels )integer coordinates) the only reason for floating point operations is the midpoint comparison part. Since it is a comparison and not a calculation, it is ok to apply mathematical operations on both sides of the equation.

So to make midpoint integer we multiply both side by 2 and to make curr_y integer, we multiply both sides by dx (because curr_y = x*m = x * (dy/dx)) Of course, the increments need to be updated as well.

Conclusion

The processing file that sums up this blog post can be found here. I did not implement all different cases the endpoints could be arranged.

It runs each algorithm 10,000 times and outputs the frame rate. It was a surprise to me that all the implementations had roughly the same fps except for the non integer midpoint which was a little faster (60 vs 65). No comment on this right now.

Experimenting with Bats

2018-07-08T23:00:00+03:00

I had some progress with my Ada Kernel Module Framework. Currently, what I have is a demo of a kernel module opening a character device.

While finishing that part, I realized that a good automatic testing strategy needs to be developed. At current stage, I do not even know which Ada features are supported. The framework is so fragile that any change can cause regression problems.

Therefore I decided to explore testing with Bats - Bash Automated Testing System.

Strategy

How does one test a framework?

My approach is to write a set of kernel modules that would demonstrate the different Ada features and Linux kernel API bindings that are supported by the framework.

Each such module is written in a way that would allow executing the following procedure:

Build the module.
Insert it into the Kernel.
Observe output through reading the message log.
Operate on the module using a special user space program or a shell script.
Cleanup

Of course, everything have to be automated to allow efficient regression testing.

Bats

Bats is a TAP-compliant testing framework for Bash. It provides a simple way to verify that the UNIX programs behave as expected.

To install Bats on Ubuntu, it is required to add a ppa repository (as of 17.10):

$ sudo add-apt-repository --yes ppa:duggan/bats
$ sudo apt-get update
$ sudo apt-get --yes install bats

It seems that there is an active and helpful community surrounding this tool and that is an important advantage for using it.

Basics

Bats tests have the form form of shell scripts with bats extension and #!/usr/bin/env bats shebag.

Each bats file is broken down into several test steps that look like:

@test "Build module" {
  run make
  [ "$status" -eq 0 ]
}

Assertions

One of the drawbacks of Bats is the fact that there is no proper, convinient way to make assertions. Fortunately, there is a third party script that makes it possible.

I found it in the following repostiroy.

In case of failure, the assertion function will display the expected and actual result.

To load the function, one have to use bat's load command. I decided to organize stuff in a way that there is a single environment script that loads all the other dependencies (currently only one):

load ../../testing-utils/environment

#! /bin/bash
#
# Environment script for Bats tests
#
#

# Assertion helpers were taken from
#    https://github.com/mbland/go-script-bash/tree/master/lib/bats
. "${BASH_SOURCE%/*}/assertions"

It is then possible to use assert_success, assert_equal and similar commands:

@test "Build module" {
   make
   assert_success
}

For other assert commands, the source code of the script is very clear and well documented. Worth opening it and reading.

Steps

The first step would be to perform some precondition tests. These do not provide more safety because if one of these fails, then some other consecutive test is ought to fail as well. The reason to have these "sanity" tests is to make it easier to debug the reason of failure.

Currently I only check if there is no loaded module with the same name as the one about to be built.

MODULE_NAME='hello'

lsmod_check() {
   lsmod | grep "$1"
}

@test "verify there is no hello module already loaded" {
   run lsmod_check $MODULE_NAME
   assert_failure
}

Next, the module is built and loaded:

@test "build module" {
    make
    assert_success
}

@test "verify creation of loadable module with the correct name" {
    run ls_check $MODULE_FILE
    assert_success
}

@test "insert module" {
    sudo insmod hello.ko
}

Now, we want to assert that the module operated correctly during it's insertion.

In the module's code, there are strategically placed printk calls which print into the kernel log:

int init_module(void)
{
    printk(KERN_ERR "Init module.\n");

    printk(KERN_ERR "Hello Ada.\n");
    adakernelmoduleinit();
    ada_foo();
    printk(KERN_ERR "%s\n", "After Ada");

    return 0;
}

In Ada code, Kernel_IO wrapper can be used:

Linux.Kernel_IO.Put_Line ("Creating device...");
Device := Linux.Device.Device_Create(
    Class       => Class,
    Parent      => Linux.Device.NONE_DEVICE,
    Devt        => Linux.Char_Device.Make_Dev(Major, 13),
    Driver_Data => LT.Lazy_Pointer_Type (System.Null_Address),
    Name        => "artiumdevice");
Linux.Kernel_IO.Put_Line ("Created device, check /dev");

The log can be read with the dmesg command. It's output is parsed and used for validation:

@test "check module init and entry into the ada part" {
    result="$(sudo dmesg -t | tail -17 | head -1)"
    assert_equal 'Init module.' "$result"

    result="$(sudo dmesg -t | tail -16 | head -1)"
    assert_equal 'Hello Ada.' "$result"
}

Finally, it is important to clean everything and bring it back to original state. This is not 100% possible since the module itself could have made some unpredictable changes to the Kernel.

@test "remove module" {
    sudo rmmod hello.ko
}

@test "clean files" {
make clean
}

Running

Here is some success example:

$ bats test.bats
✓ verify there is no hello module already loaded
✓ build module
✓ verify creation of loadable module with the correct name
✓ insert module
✓ check module init and entry into the ada part
✓ remove module
✓ clean files

7 tests, 0 failures

And here is an example where I messed the module a little bit:

$ bats test.bats
✓ verify there is no hello module already loaded
✓ build module
✓ verify creation of loadable module with the correct name
✓ insert module
✗ check module init and entry into the ada part
(in test file test.bats, line 47)
    `assert_equal 'Init module.' "$result"' failed
Actual value not equal to expected value:
    expected: 'Init module.'
    actual:   'xyz'
✓ remove module
✓ clean files

7 tests, 1 failure

Whats Next

One thing I did not demonstrate yet is how to test module's "file" interface.

My plan to do this is by implementing a helper user space program. This program will call the file operations and output the results. A bats script will drive this program.

For simple operations like read and write, bash commands could be used directly.

I will probably write more about this in some future blog post.

Migrating Blogofile to Pelican

2017-05-13T23:00:00+03:00

Since I started writing this blog in 2012, I used Blogofile as the blogging platform.

Blogofile is a static website generator, meaning that the content is written and stored in a structured format like Markdown, then it is compiled offline into a set of HTML pages that can be hosted on any web server, iregardless of it's technology. It can be even hosted cheaply on Amazon S3, which is what I am doing with this website.

While Blogofile have served me well, It's development stalled for many years, and I felt that I wanted to try something new.

After surveying modern website generators that are based on python, I decided to try Pelican.

In this blog post, I will explain how I managed to migrate from Blogofile to Pelican.

Migration

First step was to follow the standard instructions for installing Pelican and initializing a new blog.

In pelican, blog posts are stored in the content folder, so I copied all my Markdown files there.

Unfortunately, this does not work "out of the box". Some transformations on the files had to be made.

Removing ---

Blogofile uses the triple dash (---) sequences to mark the header part of the Markdown file. Pelican does not need it and in fact will renders it into the output html.

The problem is that one needs to be careful not to remove legitimate sequences like those that are part of a table definition.

artium@melon:~/Projects/nihamkin.com/content$ (master) grep -r '\-\-\-' .
./24-about-the-danger-of-merge-conflicts.markdown:---
./24-about-the-danger-of-merge-conflicts.markdown:---
...
./20-msp430-with-ili9341-display.markdown:---
./20-msp430-with-ili9341-display.markdown:---
./20-msp430-with-ili9341-display.markdown:|:------------- |:--------:| -------:|
./19-esp82266-with-ili9341-display.markdown:---
./19-esp82266-with-ili9341-display.markdown:---
./19-esp82266-with-ili9341-display.markdown:|:------------- |:--------:| -------:|
...

Here is a snippet I used to remove only the illegitimate sequences automatically:

find . -type f -exec sed -i -r '/^---$/d' {} \;

Please be very careful with this command. Run it only in the content directory.

If you run it from the wrong location, it might modify important files, so check and double check that you are running this correctly!

Replacing Categories with Tags

Blogofile uses the Categories keyword while Pelican uses Tags to describe the same thing.

This can be achieved easily with:

find . -type f -exec sed -i 's/categories:/Tags:/g' {} \;

Code

Blogofile uses the $$code(lang=Ada) syntax for syntax highlighted code blocks. Pelican's Markdown uses the ::Ada.

The following command will find all the places that use $$code.

grep -r '$$code' .

I felt that doing the transformation by hand was quicker than writing a script. It all depends on the number of results of the command above.

Static Files

My blog is hosting some static files which are not blog posts, such as images that are used in the posts.

Blogofile uses files directory which was copied as-is to the output. In the blog posts, the following syntax was used to reference the image: ![ESP8288 connected to ILI9341](/files/esp1.jpg)

In Pelican, the static files are stored in content/files. The links on the blog posts do not have to change but pelicanconf.py needs to be aware that this folder contain static files:

STATIC_PATHS = [
    'files',
    'extra',
]

Not Breaking Existing Links

One of the most important point in the migration process is not to break inbound links to the old blog posts. Breaking links will affect SEO, make a mess in google analytics and might break Disqus comments.

In pelican, it is possible to force a specific URL for a blog post.

Here is an example of how it is done. Put the following code in the header section of the markdown file:

url: 2016/11/25/writing-linux-modules-in-ada-part-3
save_as: 2016/11/25/writing-linux-modules-in-ada-part-3/index.html

I went over all my pages manually and gave each the appropriate link.

Adding a Theme

There are large number of user created themes that are available for Pelican.

With Blogofile, I developed my own look and feel based on bootstrap. As I am not a designer, it was ugly and not mobile friendly.

With Pelican, I chose to use Flex theme. The installation is straight forward so I will not repeat what is already described in the documentation.

Disqus Number of Comments

One of the things I felt was missing in Flex is a direct link from the main page to the comments section of a post, along with the number of comments.

I added this feature to Flex, as can be seen in the screenshot:

Until my pull request is approved, the changes to the templates can be viewed here.

Development vs. Production

One neat feature of Pelican is the ability to separate production and development configurations.

My use cases for this feature are:

I want to use different Disqus sitenames for production and for testing.
I do not want to use Google analytics when testing.
I want to use different site urls for testing (localhost) and production (my actual domain name). This is important because links in Pelican are generated with absolute paths based on SITEURL parameter.
I do not want to generate RSS/Atom when testing.

The production specific parameters are placed into publishconfig.py. These parameters will override the parameters in pelicanconf.py when publishing.

Here is an example of publishconfig.py from my blog:

#!/usr/bin/env python
# -*- coding: utf-8 -*- #
from __future__ import unicode_literals

# This file is only used if you use `make publish` or
# explicitly specify it as your config file.

import os
import sys
sys.path.append(os.curdir)
from pelicanconf import *

SITEURL = 'http://www.nihamkin.com'
RELATIVE_URLS = False

FEED_ALL_ATOM = 'feeds/all.atom.xml'
CATEGORY_FEED_ATOM = 'feeds/%s.atom.xml'

DELETE_OUTPUT_DIRECTORY = True

# Following items are often useful when publishing

DISQUS_SITENAME = "artiumsblog"
GOOGLE_ANALYTICS = "UA-36977465-1"

Deploying to S3

Deploying to S3 is build-in with Pelican. Simply run make s3_upload and the script will guide you through the configuration.

About the Pitfalls of Merge Conflicts

2016-12-31T14:00:00+02:00

I do not consider myself a novice software developer. And yet I still have not exceeded the list of things I can learn "the hard way".

Last month I had chance to learn about the dangers of resolving merge conflicts as well as how important is it to do a proper code review.

We use a "classic" work-flow in my work place (ancient as Greece itself). At the beginning of a development cycle, we open a branch for each feature that will be developed in that cycle. All the branches are based on the same baseline. When all developers are done, we merge the branches one by one, test the final result and it becomes the new baseline of the next development cycle. We have one specific developer whose responsibility is to merge all the features. Sometimes, when the conflict appears to be difficult, the authors of the conflicting parts are also invited to be involved.

The following code snippets are simplified representations of what happened in our code base.

The base version of a certain file contained the following lines:

if    Msg_1.Id = 1 then
   -- Do stuff 1
elsif Msg_1.Id = 2 then
   -- Do stuff 2
elsif Msg_1.Id = 3 then
   -- Do stuff 3
end if;

One of the developers added Msg_2:

procedure Proc_Msg( Msg : Msg_Type ) is
begin
   if    Msg.Id = 1 then
      -- Do stuff 1
   elsif Msg.Id = 2 then
      -- Do stuff 2
   elsif Msg.Id = 3 then
      -- Do stuff 3
   end if;
end;

-- ...

Proc_Msg( Msg_1 );
Proc_Msg( Msg_2 );

In the meantime, another developer was busy implementing a new message ID number:

if    Msg_1.Id = 1 then
   -- Do stuff 1
elsif Msg_1.Id = 2 then
   -- Do stuff 2
elsif Msg_1.Id = 3 then
   -- Do stuff 3
elsif Msg_1.Id = 4 then
   Buffer := Msg_1.Data;
   Buffer_Ready_Toggle := not Buffer_Ready_Toggle;
end if;

Please believe me that under the constraints of our system, this way of using a toggle is a correct way to send a message to another process:

if Local_Toggle /= Buffer_Ready_Toggle then
   Local_Toggle := Buffer_Ready_Toggle;

   -- Now do stuff with buffer ...

end if;

I was present during the resolution of the conflict. It was late at night, and after some manual editing of the merge result we produced following result:

procedure Proc_Msg( Msg : Msg_Type ) is
begin
   if    Msg.Id = 1 then
      -- Do stuff 1
   elsif Msg.Id = 2 then
      -- Do stuff 2
   elsif Msg.Id = 3 then
      -- Do stuff 3
   elsif Msg.Id = 4 then
      Buffer := Msg_1.Data;
      Buffer_Ready_Toggle := not Buffer_Ready_Toggle;
   end if;
end;

-- ...

Proc_Msg( Msg_1 );
Proc_Msg( Msg_2 );

Unfortunately, this code contains two problems. It was pure luck that we noticed them. Compilation error in another part of the file caused us to recheck the whole file.

Can you find the problems yourself?

The first mistake is that the line Buffer := Msg_1.Data should actually be Buffer := Msg.Data.

The second problem is a bug that will happen if both messages have Id of 4. In this case, the toggle changes will cancel each other.

The disturbing part of this story is that these problems could not have been detected by tests. Each developer's tests, even when run on the merged version would have passed successfully.

Conclusion

There are several lessons I learned from this incident.

Our work-flow has problems. We already in the process of adopting another work flow that would reduce the number of conflicts and shift the responsibility of resolving them to the developers that actually participated in the relevant code.
There should be more coordination between developers. Stand up meetings are good for this purpose, but additionally management should be involved for inter team/department coordination.
Code reviews are important, especially after merge. There are bugs that are hard to catch using tests alone.
Doing work late at night, when you are tired, is counterproductive. No matter how high the pressure is to deliver, the consequences will cost more that what you benefit. I have been bitten by this numerous times and still continue making this mistake.

Writing Linux Modules in Ada - Part 3

2016-11-25T20:00:00+02:00

View on Github Star on Github Download Pt-3

In this part, I will continue porting capabilities to the run-time

Image

The 'Image attribute is used to return the string representation of a value. This value can be integer, boolean, enumeration etc.

The run-time have a separate implementation of the attribute for each supported type. Here is the list of files from my native run-time:

/usr/gnat/lib/gcc/x86_64-pc-linux-gnu/4.9.4/rts-native/adainclude$  lsg img
s-imgbiu.adb
s-imgbiu.ads
s-imgboo.adb
s-imgboo.ads
s-imgcha.adb
s-imgcha.ads
s-imgdec.adb
s-imgdec.ads
s-imgenu.adb
s-imgenu.ads
s-imgint.adb
s-imgint.ads
s-imgllb.adb
s-imgllb.ads
s-imglld.adb
s-imglld.ads
s-imglli.adb
s-imglli.ads
s-imgllu.adb
s-imgllu.ads
s-imgllw.adb
s-imgllw.ads
s-imgrea.adb
s-imgrea.ads
s-imguns.adb
s-imguns.ads
s-imgwch.adb
s-imgwch.ads
s-imgwiu.adb
s-imgwiu.ads

These packages are not preset in our module's run-time, so when attempting to compile something like this:

package body Ada_Foo_Pack is
   procedure Ada_Foo is
      S : String := Integer'Image (42) & Character'Val (0);
   begin
      Print_Kernel (S);
   end Ada_Foo;
begin
   null;
end Ada_Foo_Pack;

Will result in compiler error:

ada_foo_pack.adb:3:22: construct not allowed in configurable run-time mode
ada_foo_pack.adb:3:22: file s-imgint.ads not found
ada_foo_pack.adb:3:22: entity "System.Img_Int.Image_Integer" not available

First Step

The first step would be to copy one of the native implementations as is.

Unfortunately, the compilation results in the following errors:

s-imgint.adb:75:10: violation of restriction "No_Recursion" at /home/artium/Projects/Ada_Kernel_Module_Toolkit/rts/gnat.adc:18
s-imgint.ads:42:07: violation of restriction "No_Default_Initialization" at /home/artium/Projects/Ada_Kernel_Module_Toolkit/rts/gnat.adc:74

No_Default_Initialization

The second violation points to the out parameter P of the following declaration

procedure Image_Integer
  (V : Integer;
   S : in out String;
   P : out Natural);

After some reading documentation and asking for help at the #ada irc channel, I understood that this violation happens because gnat.adc has both the No_Default_Initialization restriction and the Normalize_Scalars pragma.

As sparre from the irc channel explains, "Normalize_Scalars is a default initialization and you can't override the default initialization of an "out" parameter".

As explained in part 1, the file gnat.c was inherited from the bare bones tutorial. I do not know why the No_Default_Initialization restriction was added, but Normalize_Scalars looks pretty useful. Between these two I chose to remove the first one.

No_Recursion

This one is simple, the code have a recursion:

procedure Set_Digits
  (T : Integer;
   S : in out String;
   P : in out Natural)
is
begin
   if T <= -10 then
      Set_Digits (T / 10, S, P);
      P := P + 1;
      S (P) := Character'Val (48 - (T rem 10));
   else
      P := P + 1;
      S (P) := Character'Val (48 - T);
   end if;
end Set_Digits;

The following version I wrote does not use recursion:

procedure Set_Digits
  (T : Integer;
   S : in out String;
   P : in out Natural)
is
   D : Natural := 0; -- will store number of digits
begin

   declare
      T2 : Integer := T;
   begin
      while T2 /= 0 loop
         D := D + 1;
         T2 := T2 / 10;
      end loop;
   end;

   if D = 0 then
      P := P + 1;
      S (P) := '0';
   else
      for I in reverse 0 .. D - 1 loop
         P := P + 1;
         S (P) := Character'Val (48 - (T / 10 ** I) rem 10);
      end loop;
   end if;
end Set_Digits;

The exponent operator I used in the code required copying additional files to the run-time:

s-exnint.ads
s-exnint.adb

It was very tempting to rewrite this package completely, as it is not the most efficient implementation possible and does not use the Ada type system intelligently.

But as I already explained, I am guided by the principle of doing as little changes as possible to the native run-time.

The Unknown Symbol

After the changes made, compiling the module resulted in a warning:

WARNING: "system__img_int__image_integer" [/home/artium/Projects/Ada_Kernel_Module_Toolkit/hello.ko] undefined!

Trying to insert the module resulted in an error:

$sudo insmod hello.ko
insmod: ERROR: could not insert module hello.ko: Unknown symbol in module
$dmesg | tail -1
[132897.268341] hello: Unknown symbol system__img_int__image_integer (err 0)

It turns out that I forgot to add the run-time lib (libgnat.a) to the module's makefile. I am a little bit puzzled about why the previous stuff worked.

After adding "rts/adalib/libgnat.a" to _hello-y variable, the module loaded and produced the desired output:

$ sudo insmod hello.ko 
$ dmesg | tail -1
[76863.379199]  42

Enumeration

To make the Image attribute work with enumeration types I had to copy two additional files from the native run time:

s-imenne.ads
s-imenne.adb

I also had to comment two additional restrictions in the gnat.adc:

-- pragma Discard_Names;
-- pragma Restrictions (No_Enumeration_Maps);

What happens here is that the compiler will generate the strings and store them in the compiled binary.

When compiling calls to Some_Type'Image(Some_Value) the compiler will emit a call to an run-time function that will actually do the location of the correct string.

Notice that Some_Value can also be a variable, that is why the string must be resolved at run time.

Real Types

To add support for Image of real types (fixed and floating point), a bunch of additional files should be copied:

s-exnlli.adb
s-exnlli.ads
s-expllu.adb
s-expllu.ads
s-expuns.adb
s-expuns.ads
s-fatllf.ads
s-flocon.adb
s-flocon.ads
s-imglli.adb
s-imglli.ads
s-imgllu.adb
s-imgllu.ads
s-imgrea.adb
s-imgrea.ads
s-imguns.adb
s-imguns.ads
s-powtab.ads
s-unstyp.ads

I modified one of the file, s-flocon.adb and removed the import of __gnat_init_float. The Code that supposed to initialize the FPU does nothing now. I have no problems with that, so far. After all, the Linux kernel should have already did the initialization before our module is being insert.

Additionally a fixed point restriction should be lifted from gnat.adc:

--  pragma Restrictions (No_Fixed_Point);

Final Words

That is all for now. Next I will do Interfaces package and start binding functions from the Kernel itself.

Writing Linux Modules in Ada - Part 2

2016-11-05T17:30:00+02:00

View on Github Star on Github Download Pt-2

In the previous post I demonstrated that writing Linux kernel modules in Linux is not science fiction.

Yet, the produced module was extremely simple. Before improving it, some foundation needs to be added. Today I want to introduce the secondary stack.

Secondary Stack

The existence of the secondary stack allows functions to return objects of unconstrained types.

For example:

function Get_Answer return String is
begin
   return "Forty Two";
end Get_Answer;

The length of the returned String is unknown during compilation, and therefore it is problematic to allocate space for it on the primary stack. In this particular example the compiler might deduce the maximum size statically but generally, the size of the returned string could also depend on user input.

This article has a good explanation about this stack.

The secondary stack is required for 'Image attribute to work, which is in turn used in Interfaces.C package. Therefore it is important for the purpose of writing bindings to Kernel functions.

A compiler switch, as well as an attribute can control the size and type of the secondary stack.

-D`nn'[k|m]

This switch can be used to change the default secondary stack size value to a specified size nn, which is expressed in bytes by default, or in kilobytes when suffixed with k or in megabytes when suffixed with m.

The secondary stack is used to deal with functions that return a variable sized result, for example a function returning an unconstrained String. There are two ways in which this secondary stack is allocated.

For most targets, the secondary stack is growing on demand and is allocated as a chain of blocks in the heap. The -D option is not very relevant. It only give some control over the size of the allocated blocks (whose size is the minimum of the default secondary stack size value, and the actual size needed for the current allocation request).

For certain targets, notably VxWorks 653, the secondary stack is allocated by carving off a fixed ratio chunk of the primary task stack. The -D option is used to define the size of the environment task's secondary stack.

task A_Task with
  Storage_Size         => 20 * 1024,
  Secondary_Stack_Size => 10 * 1024;

As can be understood from the example, each task has it's own secondary stack. We of course need only one secondary stack as there are no tasks in our run time.

Implementation

The implementation of the secondary stack is located in s-secsta.ads and s-secsta.ads files.

I copied these files from the native rts of the installed gnat (/usr/gnat/lib/gcc/x86_64-pc-linux-gnu/4.9.4/rts-native/adainclude/s-secsta.adb) and started editing them.

I uploaded the diff here. Please Ignore the difference at the license header, I had some confusion between AdaCore and FSF versions.

The interface is quite simple. A set of function like Init, Allocate, Mark that receive an pointer to the stack in the form of an System.Address.

The reason to receive the stack as a parameter is because there might be several stacks in the system, for example a stack for each task. In this Kernel run time there are no tasks, so only one stack is required. Nevertheless, the interface can not be changed as gnat tools are expecting these exact signatures.

The implementation is even less Ada-iomatic. Untyped chunks of memory, unchecked conversions, unchecked deallocations and a lot of pointer work.

I decided not to change the implementation very much. Only remove the unnecessary stuff.

In the original code, dynamic stack is allocated on the heap as a linked list of memory chunks.

For static stack, if a stack of less that 10K is required than a memory chunk that is defined as a global in the package is used. Otherwise a memory chunk of the appropriate size is dynamically allocated.

   Default_Secondary_Stack_Size : Natural := 10 * 1024;
   --  Default size of a secondary stack. May be modified by binder -D switch
   --  which causes the binder to generate an appropriate assignment in the
   --  binder generated file.

Throughout the code, static and dynamic cases are distinguished with SS_Ratio_Dynamic. There is almost no shared code.

   SS_Ratio_Dynamic : constant Boolean :=
                        Parameters.Sec_Stack_Percentage = Parameters.Dynamic;
   --  There are two entirely different implementations of the secondary
   --  stack mechanism in this unit, and this Boolean is used to select
   --  between them (at compile time, so the generated code will contain
   --  only the code for the desired variant). If SS_Ratio_Dynamic is
   --  True, then the secondary stack is dynamically allocated from the
   --  heap in a linked list of chunks. If SS_Ration_Dynamic is False,
   --  then the secondary stack is allocated statically by grabbing a
   --  section of the primary stack and using it for this purpose.

Since currently the run time does not support any kind of dynamic allocations, I removed all the dynamic stack branches, as well as the dynamic allocation of the static stack. This renders -D flag to have no effect on the code, a 10K stack is always allocated.

It is true that dynamic allocation in the kernel can be done with kmalloc, but it is a bit tricky for me at this stage.

To allow the module code use the stack, I had to comment out a pragma restriction in the gnat.adc file.

--pragma Restrictions (No_Secondary_Stack);

This file contains a set of pragmas that are applied globally to the project. A gpr file configuration of the adc file might look like this:

package Builder is
   for Global_Configuration_Pragmas use "gnat.adc";
end Builder;

There are many other restrictions in that file that prevent using run time features that are not implemented.

After doing all this and adding a test module that demonstrates a function that return unconstrained string, I stumbled upon a strange error:

$sudo insmod hello.ko 
insmod: ERROR: could not insert module hello.ko: Invalid module format
$dmesg | tail -2
[36648.395725] module: overflow in relocation type 10 val ffffffffc04da000
[36648.395731] module: `hello' likely not compiled with -mcmodel=kernel

It turns out I had to add the "-mcmodel=kernel" flag to the compilation section in the gpr file.

Conclusion

That is it. You can try it yourself using the "Download Pt-2" link at the top of the page. The "Download Pt-X" links are snapshots of the git repository that are compatible with the appropriate blog post. So even as I continue to improve the toolkit, readers will still be able to play with the original version.

My next steps will be to add 'Image and Interfaces.C. And after that I will be able to bind some kernel functions.

Stay tuned :)

Writing Linux Modules in Ada - Part 1

2016-10-23T13:30:00+03:00

View on Github Star on Github

In the following series of blog posts I will document my attempts to write Linux modules using the Ada language.

I am not a professional and my knowledge of the Linux kernel and gnat is not comprehensive. Please pardon me for any inaccuracies I make.

Introduction and Motivation

Linux module is a binary that can be dynamically loaded and linked into the Kernel. One major use case is device drivers. Traditionally Linux modules are written in C and built using kbuild, an elaborate build system that is also used to build the kernel itself.

Code written for a module runs in the kernel with the privileges of the kernel, meaning that a programming error can cause anomalies such a system reboot, memory corruption of applications, data corruption of the hard disk and more.

Ada is a language that was designed specifically for embedded safety critical applications which makes it more suitable for writing device drivers.

Preparations

First, I investigated the current process of writing and building modules. An example tutorial can be found here.

In an nutshell, it is required to create a file that looks like this:

/*  
 *  hello-1.c - The simplest kernel module.
 */
#include <linux/module.h>   /* Needed by all modules */
#include <linux/kernel.h>   /* Needed for KERN_INFO */

int init_module(void)
{
    printk(KERN_INFO "Hello world 1.\n");

    /* 
     * A non 0 return means init_module failed; module can't be loaded. 
     */
    return 0;
}

void cleanup_module(void)
{
    printk(KERN_INFO "Goodbye world 1.\n");
}

Additionally, a mkefile that looks like this:

obj-m += hello-1.o

all:
    make  -C /lib/modules/$(shell uname -r)/build M=$(PWD) modules
clean:
    make  -C /lib/modules/$(shell uname -r)/build M=$(PWD) clean

Calling make will produce a .ko file which is the actual loadable module. It can be loaded and removed using the insmodule and rmodule.

The difference between a .ko and .o file is the additional memory sections added by the linker.

The Strategy

It is very tempting to abolish the cumbersome kbuild system and use a gpr file for the whole compilation and linkage. Unfortunately the makefiles that kbuild is made of are too complex for me to understand, and I was able to imitate everything it does to produce the ".ko" file. Additionally, it is not safe to work around the kbuild system as the internal details might change it future versions of the kernel.

So instead, I envision the following strategy. All the "logic" of the module will reside in Ada functions that will be compiled into a static library. The main file will be written in C and will consist of wrappers (eg init_module, cleanup_module) that will call the Ada functions. We will tell kbuild about the existence of the Ada library, so it will link it into the module, and everyone will be happy.

Hello World with Ada

The Code

The Ada files contain a function which always return a constant 42. pragma Export is used to export the ada_foo symbol.

package Ada_Foo_Pack is

   function Ada_Foo return Integer;

private
   pragma Export
      (Convention    => C,
       Entity        => Ada_Foo,
       External_Name => "ada_foo");
end Ada_Foo_Pack;

:::ada
package body Ada_Foo_Pack is

   Ultimate_Unswer : Integer := 0;

   function Ada_Foo return Integer is
   begin

      return Ultimate_Unswer;

   end Ada_Foo;

begin

   --  This line of code will run during elaboration
   --
   Ultimate_Unswer := 42;

end Ada_Foo_Pack;

The C file performs the elaboration of Ada libraries by calling adakernelmoduleinit, and then calls prink with the value returned by the Ada function.

#include <linux/module.h> /* Needed by all modules */
#include <linux/kernel.h> /* Needed for KERN_INFO */

extern void adakernelmoduleinit- (void);
extern int ada_foo(void);

int init_module(void)
{
    adakernelmoduleinit();
    printk(KERN_ERR "Hello Ada %d.\n", ada_foo());

    return 0;
}


void cleanup_module(void)
{
    printk(KERN_ERR "Goodbye Ada.\n");
}

So far nothing too complicated. However when building the module, kbuild complaints about many missing symbols. These missing symbols are part of the "Run Time". The calls to these methods are either produced by a compiler (for example delay 0.1; will implicitly call functions from the calendar package) or produced by the binder (initialization of the run time).

Normally, these functions can be found in libgnat.so, but this dynamic library is not available in the kernel space.

I also tried to tell kbuild to statically link with libgnat.a, but it turns out that libgnat.a is referencing other symbols which are supposed to be located in further libraries which complicates everything.

On top of this, linking the standard run time to the kernel does not make sense as it uses system calls (for example to open a file) which are available only in user space.

The Run Time

The solution is to build a custom, degraded run time. Such run times are also known as "Zero Footprint".

To achieve this I followed "Ada Bare Bones" tutorial that was written by Luke A. Guest.

To summarize, for building a run time, you will need adainclude and adalib directories. The first one will contain a copy of some of the files from the original run time, as well as system.ads, gnat.ads and additional custom packages. The second directory will contain the compiled library libgnat.a.

When building the kernel module library, every relevant tool needs to be called with the "--RTS=" flag that specify the path to the directory that contains the above two.

Deprating a little bit from Luke's tutorial, I did some modifications to the directory structure, the compiler flags, removed the console package and changed the last chance handler into a null function.

The final directory structure would look like the following diagram. I have uploaded the complete project to githhub, and you can find this version of the project by looking for the blog-post-pt-1 git tag.

Download Pt-1

.
├── gnat.adc
├── kernel_module_lib.gpr
├── lib
├── main.c
├── Makefile
├── obj
├── rts
│   ├── adainclude
│   │   ├── ada.ads
│   │   ├── a-unccon.ads
│   │   ├── a-uncdea.ads
│   │   ├── gnat.ads
│   │   ├── g-souinf.ads
│   │   ├── interfac.ads
│   │   ├── last_chance_handler.adb
│   │   ├── last_chance_handler.ads
│   │   ├── s-atacco.adb
│   │   ├── s-atacco.ads
│   │   ├── s-maccod.ads
│   │   ├── s-stoele.adb
│   │   ├── s-stoele.ads
│   │   └── system.ads
│   ├── adalib
│   ├── gnat.gpr
│   └── obj
└── src
    ├── ada_foo_pack.adb
    └── ada_foo_pack.ads

Conclusion

The example module I have written is very simple proof of concept. The Ada part does not even use any of symbols exported by the kernel. In future articles I will make an attempt to write a usefull kernel module that actually does stuff.

Connecting MSP430 with ILI9341 TFT Display

2016-04-30T20:00:00+03:00

In the previous post I described how to connect ESP8266 with an ILI9431 TFT display using the Adafruit library. Texas Instruments' MSP430 is another popular low cost 3.3v MCU used by the maker community. It has Energia which is an IDE forked from and compatible to the Arduino IDE. Therefore porting the Adafruit library to work with it should be almost trivial.

The first step is to obtain an MSP430 launchpad. You will need one that come with an MCU that has at least 16Kb of FLASH, I will explain this requirement later in this post. The currently sold 9.99$ value line version which comes with msp430g2553 MCU is sufficient. Older revisions of this launchpad might come with MCUs that does not have enough space.

After obtaining the launchpad, download and install Energia. You might want to try to upload and run some of the basic examples before advancing forward.

Another requirement is to enable hardware UART by rotating jumpers on the launchpad as explained with this tutorial. This will enable receiving debug messages from the graphicstest sketch.

As Energia is an Arduino-like environment, therefore not many changes to the library were required. The graphicstest.ino example sketch had to be modified by excluding textTest function from the sketch when compiling for msp430g2553. This is because this MCU has 16Kb of ROM memory, while the full sketch size is ~19Kb. There are MCUs with even lower ROM sizes (eg msp430g2452 which has 8Kb), if anyone would like to try this sketch, it will be required to remove more test function to reduce the size even further.

Now download the modified library and place it in the library folder of Energia. If you are cloning the repo, make sure to checkout "msp430-support" branch. After restarting Energia, you will be able to see the Adafruit_ILI9341 examples in the Examples list.

You will also need to download and place the Adafruit_GFX_Library.

Make the following connections. Those connections are specific to msp430g2553 launchpad. Other launchpads in this family have different pins for hardware SPI, so be careful and look it up in the reference.

ILI9341		MSP430
SDD/MISO	⇔	P1.6
LED	⇔	3.3V
SCK	⇔	P1.5
SDI/MOSI	⇔	P1.7
DC/RS	⇔	P2.1
RESET	⇔	3.3V
CS	⇔	P2.2
GND	⇔	GND
VCC	⇔	3.3V

DC and CS can be any of the GPIO pins, yet they need to be consistent with the values passed to the library constructor.

Here are the results of the benchmark, textTest did not run therefore its results are 0.

ILI9341 Test!
Display Power Mode: 0x9C
MADCTL Mode: 0x48
Pixel Format: 0x5
Image Format: 0x0
Self Diagnostic: 0xC0
Benchmark                Time (microseconds)
Screen fill              3986432
Text                     0
Lines                    6518272
Horiz/Vert Lines         345088
Rectangles (outline)     235520
Rectangles (filled)      8283648
Circles (filled)         2135552
Circles (outline)        2848768
Triangles (outline)      2067456
Triangles (filled)       3433984
Rounded rects (outline)  1003008
Rounded rects (filled)   9315840
Done!

It is also possible to use software SPI the example sketch has a commented section that describes how to perform this. Being said, this option is extremely slow and I do not advice using it with MSP430.

Connecting ESP8266 with ILI9341 TFT Display

2016-03-04T15:00:00+02:00

ESP8266 was popular with hobbyists for a long time. Recently I decided to join the hype and bought this module along with a super cheap color TFT display (ILI9341). In this post I will describe the process of connecting those two, using already available code written for Arduino.

First step is to set up Arduino IDE to work with ESP8266.

There is an excellent tutorial on Sparkfun's website that explains how to do this.

Once Arduino IDE is setup, you need to get the Adafruit ILI9341 library. Currently version 1.0.1 is available through Arduino IDE's "manage libraries" interface. Unfortunately this version is broken for ESP8266.

Update 2019: There is a faster library with more features called TFT_eSPI and you can use it instead. See explanation in the follow up blog post.

Until this is fixed, you have to get it directly from the Adafruit repository. One can git clone it or download the zip. Put it into the arduino library folder. The exact location is OS and IDE version depndent.

After this, fire up the IDE. If previous step was performed correctly, Under the file->examples sub-menu you should be able to see Adafruit ILI9341 menu. Choose the graphicstest example and try to compile it. If it compiles fine, you are ready for wiring.

Wiring is a bit tricky. First look at the diagram of the nodemcu v3 board:

Make the following connections:

ILI9341		ESP8266
SDD/MISO	⇔	D6
LED	⇔	3.3V
SCK	⇔	D5
SDI/MOSI	⇔	D7
DC/RS	⇔	D1
RESET	⇔	3.3V
CS	⇔	D2
GND	⇔	GND
VCC	⇔	3.3V

Make sure you connect all display's pins. Some tutorials do not mention RESET and LED pins and it took me some time to figure out that these can not be left floating.

Now you need to alter the example sketch to reflect the fact that we chose D1 and D2 for CS and DC. Edit the configuration lines to be:

// For the Adafruit shield, these are the default.
#define TFT_DC D1
#define TFT_CS D2

Now connect the board to the computer, build and upload. You should be able to see a a test pattern start running on the TFT screen.

Unfortunately, at some point the test pattern will freeze or even reset the device before the program completes the execution.

If you open serial monitor, you will see something like this:

���ԅu�D�ILI9341 Test!
Display Power Mode: 0x9C
MADCTL Mode: 0x48
Pixel Format: 0x5
Image Format: 0x0
Self Diagnostic: 0xC0
Benchmark                Time (microseconds)
Screen fill              2259125
Text                     161736
Lines                    
Soft WDT reset

ctx: cont 
sp: 3ffef8d0 end: 3ffefbd0 offset: 01b0

>>>stack>>>
3ffefa80:  016e3600 00000001 3ffeea58 000007e0  
3ffefa90:  00000080 00000008 3ffeea58 402032f8  
3ffefaa0:  3ffeea9c 00000001 3ffeea58 0000006c  
3ffefab0:  0000006d 0000002a 3ffeea58 40203378  
3ffefac0:  3ffeea9c 000000f6 3ffeea58 402037b2  
3ffefad0:  000000f7 000000f6 3ffeea9c 402027ab  
3ffefae0:  016e3600 00000001 3ffeea58 000007ff  
3ffefaf0:  0000006c 000000f6 3ffeea58 40203876  
3ffefb00:  016e3600 00000001 3ffeeb7c 00000000  
3ffefb10:  00000000 000000f0 3ffeea58 000000ef  
3ffefb20:  0000006c 000000ef 0000005f 402046e1  
3ffefb30:  00000000 00000001 000007ff 00000085  
3ffefb40:  3ffeea58 000000f6 402044b8 00000144  
3ffefb50:  000000f0 000007ff 3ffeea58 00cb82d4  
3ffefb60:  000000ba 000007ff 3ffeea58 40201c40  
3ffefb70:  0000013f 000000ef 00000140 000000ef  
3ffefb80:  000000ef 3ffeeba8 402044b8 3ffeebb0  
3ffefb90:  3ffeeb7c 00000018 4022d52d 3ffeeba8  
3ffefba0:  00000000 000000c0 3ffeeb7c 40202398  
3ffefbb0:  3fffdc20 00000000 3ffeeba0 402044f6  
3ffefbc0:  00000000 00000000 3ffeebb0 40100114  
<<<stack<<<

The problem here is that ESP8266 has a watchdog set to one second. The lines test pattern takes longer than one second to execute and therefore the watchdog resets the processor.

The ILI9341 example is already "feeding" the watchdog using the yield(); call. Unfortunately, on the ESP8266, some code between the yield(); calls still takes more than one seconds (this code was written with Arduino in mind). To make this example run, you will have to add additional yield(); calls. Use trial and error method to figure out where to add those.

Disabling the watchdog is also a viable options, but I was not able to figure out how to do it. If you find a way to do this, write me through the comments.

Trying Ada Bindings for X11

2015-09-01T17:00:00+03:00

Ada bindings for X11 were written Intermentics company and sponsored by Ada Joint Program Office (AJPO).

While intemetics has long been gone (looks like the it's domain is owned by L3 now), and AJPO was closed in 1998, the bindings are still around.

The "latest" version can be downloaded from this page on adapower web site (the download link is at the bottom named x11ada)

I was surprised how smooth was the process of building and running a demo program. The following tutorial will walk through this process.

Setup

The tools I used for this tutorial are gnat 4.9.3 and gprbuild 2014.
Create a directory for your project. I called it "ada11_testing".
Download "x11ada_v1.30.tar.gz" from the link provided above. Inside you will find "ada" directory. Copy this whole directory to "ada11_testing".
Create a directory called "obj" inside "ada11_testing".

Project File

Create "test.gpr" file with the following contents:

project test is
  for main use ("test");

  for Source_Dirs use ("./**");

  for Object_Dir use "obj";

  for Exec_Dir use ".";

  package Linker is
     for Default_Switches("Ada") use ("-lX11");
  end Linker;

  package Compiler is
     for Default_Switches("Ada") use ("-gnateE", "-gnat2012"); 
  end Compiler;

  for Languages use ("Ada", "C");

end test;

Code

Being thin bindings, every C function of X11 library has an equivalent Ada function. Therefore example programs, documentation and tutorials that are aimed at C are suitable for Ada programmers as well.

I used example program from the following article of Linux Magazine.

with Interfaces.C;
with X;
with X.Xlib;

procedure Test is

   use type Interfaces.C.Int;
   use type Interfaces.C.Unsigned;
   use type X.Xlib.XDisplay_access;

   Cant_Open_Display_Error: exception;

   Display          : X.Xlib.XDisplay_access;
   Screen_Num       : Interfaces.C.Int;
   Win              : X.Window;
   Graphics_Context : X.Xlib.GC;
   Report           : aliased X.Xlib.XEvent;

begin

   Display := X.Xlib.XOpenDisplay( null );

   if Display = null  then
      raise Cant_Open_Display_Error;
   end if;

   Screen_num     := X.Xlib.DefaultScreen(Display);

   Win := X.Xlib.XCreateSimpleWindow
     (Display      => Display, 
      Parent       => X.Xlib.RootWindow(Display, Screen_Num), 
      XX           => 50, 
      Y            => 50, 
      Width        => 200, 
      Height       => 200, 
      Border_Width => 0, 
      Border       => X.Xlib.BlackPixel(Display, Screen_Num), 
      Background   => X.Xlib.WhitePixel(Display, Screen_Num));

   X.Xlib.XMapWindow(display, Win);

   X.Xlib.XSelectInput(Display, Win, X.StructureNotifyMask);

   loop
      X.Xlib.XNextEvent(Display, Report'access);

      exit when Report.Event_Type = X.MapNotify;
   end loop;

   Graphics_Context := X.Xlib.XDefaultGC(Display, Screen_Num);

   X.Xlib.XSetForeground(Display, Graphics_Context, X.Xlib.BlackPixel(Display, Screen_Num));

   X.Xlib.XDrawLine(Display, X.Drawable(Win), Graphics_Context, 10, 10, 190, 190);
   X.Xlib.XDrawLine(Display, X.Drawable(Win), Graphics_Context, 10, 190, 190, 10);

   X.Xlib.XSelectInput
     (Display, 
      Win, 
      Interfaces.C.Long( Interfaces.C.Unsigned( X.ButtonPressMask) or X.ButtonReleaseMask)
     );

   loop
      X.Xlib.XNextEvent(Display, Report'access);

      exit when Report.Event_Type = X.ButtonRelease;
   end loop;

   X.Xlib.XDestroyWindow(Display, Win);
   X.Xlib.XCloseDisplay(Display);

end Test;

Building with gprbuild:

gprbuild -P test.gpr

And that is all it takes to use 20 years old software package!

Range Constrained Types in C++

2014-09-05T22:00:00+03:00

One of the first things a new Ada programmer will learn is the ability to define constrained type. Which means that one can restrict the values that can be assigned to a variable of this specific type.

For example:

subtype Positive is Integer range 1 .. Integer'Last;

Being a subtype of Integer, Positive is fully compatible with it and can be used wherever Integer is applicable. Ada's runtime system however, will guard against assignments of values lower than 1 and will dispatch a runtime error whenever this happens.

This feature enhances type safety and reduces probability for bugs. For example, one can safely divide an Integer by Positive without fear of dividing by zero.

C++ does not have this feature. There is an unofficial Boost library claiming to implement it. However reading the implementation I found out that constrained_value is not a proper subtype of value_type. There are examples where instances of the first, can not substitute the other:

void f(bounded_int<int, 0, 100>::type x) { ... };

...

bounded_int<int, 0, 100>::type x = 5;
x++; // Compilation error: operator not defined.
f(x); // Compilation error: casting operator not defined.

With these limitations, this feature becomes less useful.

Therefore, as an exercise, I decided to implement this feature on my own. The goals of my implementation were:

Proper subtyping of the base type.
Smallest possible performance and memory penalty.
Simple interface.
Extensive automatic testing.

How to Use

The code itself can be found on github. Just grab the file named subtype_range_constrained.h and add it to your project.

Here is a simple usage example:

typedef ct::RangeConstrained<short, 1, 12> positive;

int main(void) {
    positive p = 12;
    int x = -3;

    p = p + x;  // Ok
    p = x;      // Exception!
}

Template Argument Deduction

2014-05-04T23:00:00+03:00

With automatic template deduction, it is possible to omit the template argument when instantiating a template function. The compiler will deduce the type from the actual parameter sent to the function. This is of course assuming there are no ambiguities.

For example:

template<class T>
T max(T x, T y) {}

int main(void) {
    return max(0,1);       // equivalent to max<int>(0,1);
}

As I have found recently, there is more to this feature than meets the eye.

Assume that we want to write a function that would merge two std vectors. A usage example would look like this:

vector<int> v1(5,10), v2(4,12),v3;

v3 = merge_vectors(v1,v2);

Since vector is a class template, the following signature would not compile:

vector merge_vectors(vector v1, vector v2) {
     // ...
}

error: invalid use of template-name ‘std::vector’ without an argument list

We do not want to create a separate function for each vector instantiation we use in our program . The solution is to use a function template:

template<class T>
vector<T> merge_vectors(vector<T> v1, vector<T> v2) {
     // ...
}

The deduction here is more complex than in the first example. T is deduced not by the parameters sent to the function, but by the generic argument used to instantiate those parameters. This is important because we want to constraint the parameters to be vectors, we just do not care what kind of vectors they are.

How to Enable Logging of Chat Sessions in ERC

2013-12-04T14:00:00+02:00

ERC is an irc client that runs inside emacs. I sometimes use it to ask question or read interesting discussions.

It is very nice to have the code I am working on and some interesting discussion side by side on the screen.

Sometimes I read something interesting and later want to recall what exactly I read.

For such cases ERC provide an ability to log chat sessions that the user is participating in. It is off by default, so it must be enabled in the configuration.

I have the following configuration in my .emacs, it took me some time to figure how to do it correctly, so I hope this information will help somebody else who is struggling as well:

;; it is not possible to set erc-log-mode variable directly 
(erc-log-mode) 
;; The directory should be created by user.
(setq erc-log-channels-directory "~/.erc/logs/")
(setq erc-generate-log-file-name-function (quote erc-generate-log-file-name-with-date))
(setq erc-save-buffer-on-part nil)
(setq erc-save-queries-on-quit nil)
(setq erc-log-write-after-insert t)
(setq erc-log-write-after-send t)

There are few tricky thing to consider here. Firstly, the logs directory should exist, if ERC does not find the provided path, it will do nothing.

Secondly, the call to erc_log_mode enables logging. It can be confusing because unlike other configuration options, it is a call to a function. As the documentation says:

Erc Log Mode:
Non-nil if Erc-Log mode is enabled.
See the command `erc-log-mode' for a description of this minor mode.
Setting this variable directly does not take effect;
either customize it (see the info node `Easy Customization')
or call the function `erc-log-mode'.

There are more ERC related customization options available. It is possible to see them with the following emacs sequence:

M-x customize-group RET erc RET

This will display a nice GUI-like editor for those options and also will group them according to functionality.

Making Legacy Software Work On Windows

2013-11-09T14:00:00+02:00

It sometimes happens that programs written for old versions of Windows will not work on new versions even with compatibility mode enabled. It is not uncommon for such program to crash and show some cryptic message that even a person of technical background can not understand. Sometimes only a simple fix is required, like providing a missing DLL or creating a registry key, yet the program will not tell us what it actually expected to find. Recently I have discovered a technique that can help debug such situations. It might surprise the reader the the solution comes from the Linux world.

Wine is a tool that allows running Windows programs on Linux. It does this by providing a Linux implementation of Window's API and system calls. One neat feature of Wine is the ability to output which system calls are being called during an execution of a program.

Example

I will demonstrate how to get advantage of this feature with a game called "Janes F-15". This is a flight simulator released in 1998 for Windows 95.

The problem I had, was not with the game itself but with it's installer. The installer would say "ERROR, Copying files" right after it began it's operation. Which files it was unable to copy it did not specify.

Finding which file is missing can be done by reading the debug messages. The debug messages are enabled by setting the WINEDEBUG environment variable before running the program:

$ export WINEDEBUG=warn+all

The number of messages produced by even a boilerplate program is very big, the console will spit text long after the application is closed, and it will be hard to determine cause and effect behavior. Therefore different "debug" channels exist. In the example above I have used the "all" channel and limited it only to warnings. Other scenarios would require other channels. In the following link a complete documentation of all the channels can be found.

So I set the environment variable and run the installer. Just before clicking the last "next" button of the wizard (which causes the installation process to actually start), I brought the the console to the front. Clicking on the final "next" button caused some more messages to be printed into the console, with the most relevant messages being printed last:

It is now clear what the problem is.

warn:ntdll:NtQueryAttributesFile L"\\??\\C:\\windows\\system32\\EAREMOVE.EXE" not found (c0000034)
warn:file:OpenFile (C:\windows\system32\EAREMOVE.EXE): return = HFILE_ERROR error= 2
warn:ntdll:NtQueryAttributesFile L"\\??\\C:\\Janes\\F15\\EAREMOVE.EXE" not found (c0000034)
warn:file:OpenFile (C:\Janes\F15\EAREMOVE.EXE): return = HFILE_ERROR error= 2
warn:ntdll:NtQueryAttributesFile L"\\??\\E:\\EAREMOVE.EXE" not found (c0000034)
warn:file:OpenFile (.\EAREMOVE.EXE): return = HFILE_ERROR error= 2

The installer searches for a file called "EAREMOVE.EXE", first in system32 directory, then inside the directory where the game is about to be installed, and finally on the installation disk. I do not know what is the logic behind this behavior and why this file was missing.

Creating an empty file named "EAREMOVE.EXE" in the system32 directory solved the problem and allowed the installer to continue. The game was installed correctly and I was able to run it.

Combining ls and cat Commands

2013-08-17T04:00:00+03:00

Very often, when exploring contents of directory trees, I find myself confused between the the "ls" and "cat" commands and mistakenly type one instead of the other. Both these commands are used pretty often and conceptually they have similar meanings. One says "print out a content of directory". The other says "print out the content of a file".

Therfore, it is natural for these commands to be combined into a single one. So, I decided to write a shell script that would do exactly this.

The following shell script will choose to run "ls" or "cat" depending on the type of the parameter (file or directory).

#!/bin/sh

## lc.sh
## Artium Nihamkin
## August 2013
## 
## This script will to run "ls" or "cat" depending the type of the input.
##
## Examples where "ls" is invoked:
##    ./lc.sh file.txt
##    ./lc.sh
## Examples where "cat" is invoked
##    ./lc.sh ~/directory.name
##    ./lc.sh does.not.exist
##    echo "xxx" | ./lc.sh 
##

# If stdin present, assume user meant cat
#
if [ ! -t 0 ] 
then
    cat "$@"
    return
fi

# Find the argument that is file/directory name and test it
# to find out if it is an existing directory. Other arguments are 
# options and thus will begin with an "-".
#
for v in "$@" 
do
    if [ '-' != `echo "$v" | cut -c1 ` ] 
    then
        if [ -d "$v" ]
        then
            # A directory, use "ls".
            #
            ls "$@"
            return
         else
            # Not a directory, use 'cat'.
            # If this is not a file or a directory then cat will
            # print an error message:
            # cat: xxx: No such file or directory
            #
            cat "$@"
            return
        fi
    fi  
done

# No file name provided, assume the user is trying to ls 
# the working directory
#
ls "$@"

Notice that running this script without a file/directory name parameter is equal to a plain "ls" which will list the working directory, but if you provide an stdin input it will assume the user is expecting the behavior of a "cat".

Assuming that you have put this script in /my/pah/to/lc.sh path, you can now add an alias to this command:

alias lc="/my/path/to/lc.sh"

Add this line into ~/.bashrc and it will run every time you start an interactive shell.

$ ls
lc.sh
$ lc
lc.sh
$ mkdir test
$ echo "abc" > test/test.txt
$ lc -l test
total 4
-rw-rw-r-- 1 artium artium 4 Aug 17 03:47 test.txt
 lc test/test.txt
abc
$ echo xyz | lc
xyz

You can even go as far as adding an alias for the "ls" and "cat" commands, thus overriding them. This will allow you to call "ls" on files and "cat" on directories and still get a desired effect.

alias ls="/my/path/to/lc.sh"
alias cat="/my/path/to/lc.sh"

There two thing you need to know here :

If for saome reason you will need to run the original command, you can type an "\" before it's name (for example "\ls") and this will disregard any aliases and run the original command.
Aliases are not expanded inside non-interactive shell scripts. So if you write "ls" inside a script, it will call the original "ls". This is good bacuase we don't want to break existing scripts that might be relaying on some esoteric behavior of "ls" or "cat" which is not imitated in my script. This is true as long as those scripts did not run shopt -s expand_aliases which causes the expansion of aliases inside non-interactive scripts. Pretty rare and low risk, yet good to be aware.

OpenNI2 Samples and Dynamic Library Problems

2013-08-02T17:00:00+03:00

While experimenting with the OpenNI2 samples, I sometimes would get the following error when trying to run one of the executables:

~/OpenNI-Linux-x86-2.2$ Samples/Bin/SimpleViewer
Samples/Bin/SimpleViewer: error while loading shared libraries: libOpenNI2.so: cannot open shared object file: No such file or directory

The dynamic library file libOpenNI2.so is located in the Bin folder alongside the executable. One would expect that like with DLLs on Windows, the dynamic linker have the executable's directory in it's search path.

Well, it turns out that for Linux's linker it is not the case. It searches in the working directory but not in the directory of the executable.

This can be easily fixed by copying the library into a directory that is on the search path of the linker. According to this guide, the convention is to put libraries in /usr/local/lib. So all that needs to be done are the following actions:

~$sudo cp "OpenNI-Linux-x86-2.2 Samples/Redist/libOpenNI2.so" /usr/local/lib
~$sudo ldconfig

How to Save Money in Israel

2013-04-01T16:00:00+03:00

It is always good to cut on household expenses, especially at times when product prices are rising. In this post I have gathered several tips that will help you save money monthly with only small impact on your quality of life.

Notice that this post is intended to help Israeli residents and will not be as useful for visitors and tourists.

Cellphone Bills

Thanks to the cellphone market reform the prices today are quite fair. Yet many people are not aware of this and still pay for the older plans. New physical and virtual carriers offer "unlimited plans" for a price of 90-100 NIS per month. Some of the older carriers offer similar plans as well.

If you do not use the phone very often you can reduce the bill even further by choosing a "pay per minute" plan. You need to look at your previous bills and see how many minutes you talk each month. Then calculate the price you will pay for these minutes in the "pay per minute" plans and decide which plan is cheaper for you.

I recently switched my family to one of the new companies. My mother used to pay 100-110 NIS per month for her calls. Now she pays 10 NIS for her plan that grants her 60 minutes and 60 sms per month and on average she pays additional 20 NIS for the extra minutes she talks. That is an instant 70 NIS saving.

Telephone Bills

My family still keeps a Bezeq land-line at our home, but in my opinion there is absolutely no reason for this. Landlines are rarely used these days and 50 NIS just for the subscription is a too high price to pay when you have mobile unlimited plans. Also notice that contrary to a common belief, according to a new law, if you cancel your land-line you will still be able to keep Bezeq as your Internet infrastructure.

For those that want to keep the phone number so people will be able to call, there is a light subscription plan that is called "Kav Kal" ("קו קל") and it costs 25 NIS per month.

Television

Nowadays we do not spend as much time watching TV as we used to. The Internet and Smartphones/Tablets have taken over the entertainment spot in our life. If you still want to be able to watch the news on the Israeli channels, you can buy an Idan+ receiver with one time investment of ~200NIS. With most Idan+ receivers you will also be able to play movies from a plugged-in disk on key.

Internet

The key idea here is to hassle. First, research on the Internet for the market price of the speed you are interested to subscribe to. There are several sites (for example kamaze.co.il) on which people share the price they got from their company (either infrastructure or provider). You will notice that the lowest price for each speed is shared among all companies. Next, call for your company and stick to that price you want. When they tell you that X is the price they can give you and nothing can be done to lower it, tell them that you did your research and know the "market price". Then they will probably bounce you from one representative to a more "senior" representative which will offer a lower price and maybe some "extras". Do not be tempted to take the "extras", stick to your price and if they will not bounce you any more, ask to disconnect immediately. They will bounce you one last time to thing called "customer retention" (shimur lakohot) and there you will get the price you want.

Credit Cards

Some big chain stores offer a consumer club cards in a form of credit card. These card are usually free for the first year, but after that you have to pay a monthly fee. If you use these cards infrequently, it is good to reevaluate their benefit and maybe cancel some of them. On the other hand, if you use such card frequently it is good idea to cancel the ordinary credit card you got from your bank.

Groceries

Comparing prices is time consuming and not always worth the money that can be saved. My advice here is to compare prices on line. There are several sites (for example mysupermarket.co.il) available that will allow you to input your desired basket of groceries and they will show you the cheapest chain store for your basket. Some will even recommend similar products of cheaper brand to help you save more money.

Conclusion

It is hard to asses the savings that can be gained for the average family, but for amusement lets wildly guesstimate it:

Here is what a family of 3 that did not have any of the bundle subscriptions (ie. TV + Phone + Internet bundles) can save a month:

Switching cellphone plans: 3 × 70 NIS
Canceling landline: 50 NIS
Canceling television plan: 200 NIS
Hassling over the Internet price: 20 NIS + 40 NIS
Canceling 2 credit cards: 2 × 10 NIS
Saving 10% on groceries: 0.1 × 1500 NIS

Total: 690 NIS

Spartanization Plug-In For Eclipse

2013-03-09T02:00:00+02:00

I would like to share an Eclipse plug-in that I have recently developed. It possible to install it from the following update site: http://update.nihamkin.com/spartan/.

If you are not familiar with "spartan programming", you can read about it on this Jeff Atwood's post or here for more detailed description.

My plug-in makes it possible to perform some of the spartanizations automatically. The following refactorings will be added into the Eclipse's refactor menu:

Convert Conditional to Ternary - when possible, this refactoring will convert short if-else statements into ternary statements thus reducing the number of lines.
Eliminate Redundant Equalities - will remove true or false from expressions which explicitly compare a Boolean with true or with false.
Shortest Conditional Branch First - will switch between if and else branches of if-else statements to make sure that the shortest branch is always first.

There are two ways to apply the refactoring; either on the selected text, or if no text is selected, on a whole project (depends on the file active in the editor).

The plug in is not polished yet. Some edge cases should be included. For example it is desirable that "Eliminate Redundant Equality" will also eliminate comparisons to null or that all refactoring that add parentheses will add them only when needed (currently "a != b" will be refactored into "!(a)" ).

Until I find time to fix these, feel free to take a look at the source code and maybe fork it and adjust it for your own needs. You can use it as an example of how to work with the refactoring framework of Eclipse.

Start New Blogofile Post With a Single Command

2013-02-22T20:00:00+02:00

I wrote a bash script which should allow starting blogofile blog posts more conveniently. This will set up everything needed with a single command.

This script will perform:

Create a file in the _posts directory. The file name will be prefixed with the next number in the sequence (personal choice to enumerate the markdown files for better organization)
The file will contain a starting template for a markdown blog post (a header with placeholders for title, date etc.)
Add the file to source control.
Open the file in emacs.

The script is:

#!/bin/bash


POSTS_DIR=_posts
FILE_EXT=markdown
TEXT_EDITOR=emacs
SOURCE_CONTROL_CMD="git add"

read -d '' TEMPLATE <<"EOF"
Tags: 
date: 
title: 
draft: True

EOF


if [ $# -le 0 ]; then
    echo "Usage: $0 postfix-name"
    exit
fi
if [ $# -ge 2 ]; then 
    echo "Warning: whitespaces in the name will be ignored" 
fi


cd $POSTS_DIR
last=`ls *.$FILE_EXT | sed 's/^\([0-9]*\).*/\1/g' | sort -n | tail -1`
next=`expr $last + 1`
next_name=${next}-$1.$FILE_EXT
echo "Creating new file ($next_name)..."
echo "$TEMPLATE" > $next_name
echo "Adding file to source control"
( $SOURCE_CONTROL_CMD $next_name )
echo "Opening file in $TEXT_EDITOR..."
( $TEXT_EDITOR $next_name & )

The way it is currently written, it supposed to work from the main directory (one above "_posts"), but feel free to tweak it for your needs.

Do not forget to add a "_" prefix, otherwise when you build, blogofile will copy it into "_site".

How to Add Items Into Eclipse's Refactor Menu

2013-02-16T17:00:00+02:00

Recently, I was writing a plug-in that performs some refactoring (of which I will write in a separate post).

Writing the refactoring was not too complicated, but figuring out how add the menu items correctly into the Refactor menu took a lot of time.

I tried to figure this from the documentation and examples that I dug from google (some of which were incorrect), and finally paulweb515 from #elicpse-dev channel explained me how it should be done.

Here is the correct plugin.xml snippet:

<?xml version="1.0" encoding="UTF-8"?>
<?eclipse version="3.2"?>
<plugin>

<extension
         point="org.eclipse.ui.actionSets">
      <actionSet
            description="Spartan Refactoring Actions"
            id="il.ac.technion.cs.ssdl.spartan.refactoring.actionSet"
            label="Spartanization"
            visible="true">
        <menu
               label="Refactor"
               path="edit"
               id="org.eclipse.jdt.ui.refactoring.menu">
            <separator name="undoRedoGroup"/>
            <separator name="reorgGroup"/>
            <separator name="codingGroup"/>
            <separator name="reorgGroup2"/>
            <separator name="typeGroup"/>
            <separator name="typeGroup2"/>
            <separator name="codingGroup2"/>
            <separator name="typeGroup3"/>
            <separator name="spartanGroup"/>
            <separator name="scriptGroup"/>
        </menu>
        <menu
               id="il.ac.technion.cs.ssdl.spartan.refactoring.menu"
               label="Spartanization"
               path="org.eclipse.jdt.ui.refactoring.menu/spartanGroup">
            <separator name="group" />    
        </menu>

         <action
               class="il.ac.technion.cs.ssdl.spartan.refactoring.ShortestBranchAction"
               enablesFor="*"
               id="il.ac.technion.cs.ssdl.spartan.refactoring.actions.ShortestBranchAction"
               label="Shortest Conditional Branch First..."  
               menubarPath="org.eclipse.jdt.ui.refactoring.menu/il.ac.technion.cs.ssdl.spartan.refactoring.menu/group"
               style="push">
         </action>

        <!-- More actions -->

      </actionSet>
   </extension>
</plugin>

The Refactor menu tag is a copy of the original "Refactor" menu definition from org.eclipse.jdt.ui. It is important to copy all the separator definitions. For my plug-in, I have also added a new separator called spartanGroup.

The path for the Refactor menu is org.eclipse.jdt.ui.refactoring.menu.

It is possible to figure this information by looking at plugin.xml of the org.eclipse.jdt.ui plug-in. You can find it online (for example here) or you can import the source code of your eclipse build following these steps:

File -> Import
Select Plug-ins and Fragments and click next.
In the "Import From" section choose "The active target platform".
In the "Import As" section choose "Project with source folders"
In the next screen, locate your plug-in (org.eclipse.jdt.ui) and add it.
Click finish and the source code of the plug-in will be imported into your workspace.

How to Install an Acer Scanner Under Linux

2012-11-12T14:00:00+02:00

I own an old Acer szw4300U scanner. Making it work with Linux is not exactly a plug and play, but not very complex either. Here is how it is done.

Drivers

An easy way is to use SANE which is a framework that connects scanning software (frontends) with the scanners' drivers (backends).

On the following page you can find a list of supported scanners with reference to their drivers. My scanner uses a "snapscan" driver.

Both SANE and the drivers are already installed by default on most Linux distributions. To verify that SANE is installed use the following command:

~$ scanimage -V
scanimage (sane-backends) 1.0.22; backend version 1.0.22

scanimage is a defualt frontend that is bundled with SANE. Let's try to scan something with it. Do not forget to plug your scanner into the computer:

~$ scanimage
[snapscan] Cannot open firmware file /usr/share/sane/snapscan/your-firmwarefile.bin.
[snapscan] Edit the firmware file entry in snapscan.conf.
scanimage: open of device snapscan:libusb:006:002 failed: Invalid argument

The issue here is that my driver is a generic one, meaning it can handle different models of scanners depending on a provided firmware file. A firmware file has .bin extension and can be found on the windows driver installation disc. The disc usually contains several firmware files, to find out which is the firmware file for your scanner model please refer to this table at the snapscan homepage. Notice that the number after the V in the name of the file is the version number, so you might find that your disc contains an older version than the one listed on the website. Nevertheless, the firmware will still fit.

Configuration

The next step is to configure snapscan to use the firmware. For example my file is U176V042.BIN. We need to copy it into /usr/share/sane/snapscan/ (create the directory if it does not exist).

Now open /etc/sane.d/snapscan.conf and do the following:

Uncomment the line

/dev/usb/scanner0 bus=usb

and then replace

firmware /usr/share/sane/snapscan/your-firmware.bin

with

firmware /usr/share/sane/snapscan/U176V042.BIN

Frontend

Finally we can test the scanner:

~$ scanimage > test.pnm
[snapscan] Scanner warming up - waiting 20 seconds.
[snapscan] Scanner warming up - waiting 20 seconds.
~$ pnmtojpeg test.pnm > test.jpeg

If you want a GUI frontend you can see the list here. I personaly use XSane and it is quite good.

Common Problems

If you get an error similar to

[snapscan] Cannot open firmware file /usr/share/sane/snapscan/U176V042.BIN

And absolutely sure that there are no typos in the conf file, I would suggest to look into the permissions of U176V042.BIN file. The permissions should allow reading for "all". Here is a reminder of how to do it:

/usr/share/sane/snapscan$ sudo chmod a+r U176V042.BIN

Strange Blogofile IO Error

2012-10-13T15:00:00+02:00

Sometimes when building a blogofile site I encounter the following IO error:

Traceback (most recent call last):
  File "/usr/local/bin/blogofile", line 9, in <module>
    load_entry_point('Blogofile==0.7.1', 'console_scripts', 'blogofile')()
  File "/usr/local/lib/python2.7/dist-packages/Blogofile-0.7.1-py2.7.egg/blogofile/main.py", line 135, in main
    args.func(args)
  File "/usr/local/lib/python2.7/dist-packages/Blogofile-0.7.1-py2.7.egg/blogofile/main.py", line 201, in do_build
    writer.write_site()
  File "/usr/local/lib/python2.7/dist-packages/Blogofile-0.7.1-py2.7.egg/blogofile/writer.py", line 53, in write_site
    self.__write_files()
  File "/usr/local/lib/python2.7/dist-packages/Blogofile-0.7.1-py2.7.egg/blogofile/writer.py", line 103, in __write_files
    t_file = open(t_fn_path)
IOError: [Errno 2] No such file or directory: './.#index.html.mako'

This error was a bit confusing. Why does it say that index.html.mako does not exist when it is open right here in my editor?

Well the editor, specifically Emacs is the cause of problem here. If you do not save your files immediately before running blogofile build there is a chance that an Emacs auto-save file will appear in your directory. The file looks like #index.html.mako# and blogofile probably have problems with the unusual file name. Once you save your files with C-x C-s the file will be gone.

Alternatively you can make Emacs save the auto-save files in one dedicated directory. Here is an emacswiki article that explains how to achieve this.

Highlighting Special Posts in Blogofile

2012-09-25T17:00:00+02:00

This simple but use-full blogofile hack will allow you to alter the style of posts which are marked as special. For example the post you are reading belongs to the "Special" category and therefore has a golden background instead of the gray default one.

This is quite straightforward hack, all you have to do is to modify the "_templates/site.mako" template (assuming version 0.7.1).

Replace the line:

<div class="blog_post">

with the following snippet:

% if "Special" in [x.name for x in post.categories]:
<div class="blog_post" style="background-color: gold;">
% else:
<div class="blog_post">
% endif

As can be seen, we check for the existence of the category name "Style" in a list that contains the names of all the categories of the current post. If this name exists we output the div element with custom style.

This is enough for simple usage, but the idea can be extended for multiple category names and styles, allowing us to style each category differently.

Add the following code at the top of "_templates/site.mako":

<%
category_styles = {"Special" : "background-color: gold;",
                   "Another Example" : "border: 3px dashed black;" }
category_names = [x.name for x in post.categories]
style = "".join([v for k,v in category_styles.items() if k in category_names])
%>

As you can see, it is possible to add more category-style pairs into the "category_styles" dictionary. Ideally we would like to put this into the _config.py file , but currently I do not know how to do it.

And finally, we need to update the div element to look like this:

<div class="blog_post" style="${style}">

Why I Chose Blogofile as my Blogging Platform

2012-09-02T01:00:00+03:00

Blogofile is the blogging engine behind this blog.

First of all, what is "Blogofile"? The definition from the original website says:

Definition: Blogofile (n): A static website compiler and blog engine, written and extended in Python.

Blogofile creates a static, full featured blog out of posts that are stored in text files. This static website then can be copied to any web server capable of serving static content. No need for PHP, python, .NET or CGI installed. This is contrary to blogs that use dynamic platforms. In these platform, pages are generated on demand, each time a user tries to access them (caching may apply).

There are some pros/cons for using static website generator for blogging. These have already been discussed on many blog posts by bloggers that switched to Blogofile. If you are interested in reading more just google it.

I will provide several reasons that bought me in. Please excuse me if those are duplicates of things that others have already noticed.

Currently this website is hosted as a personal home page my university provides for students. For security reasons, students can host only static content, so using static content is the only option I have.
When I will not be able to host my blog on my university's server, I will be able to move to any other server independently of available technology, eventually making the hosting cheaper.
Because there is no "code behind", I do not have worry about security.
Bologofile can be easily customized. It is easy to learn and setup and beginner friendly.

Hopefully in the future I will costumize Blogofile and share some experience with the readers.

About this Blog

2012-09-01T17:00:00+03:00

Welcome!

My name is Artium Nihamkin and I am a software engineer from Israel.

This is my personal blog where I plan to publish interesting and useful stuff that I encounter during my endeavor in my professional and personal life.

Some of the things you might find here are:

Articles related to software developer.
Problems and solutions for various technical problems that I have personally encountered.
Philosophical monologues.
Discoveries in the world of hobby robotics.
Links to interesting information in other blogs.

I do not expect people to follow my blog closely. My expectations are that individual posts will be discovered through various google searches.

You always can reach me by the following email: