Prostendo

A difference in error messages between GCC and Clang

Fri, 16 Nov 2018 21:10:32 +0000

C++ can be fairly complicated language and it doesn’t help that the compiler warnings aren’t always the most helpful. As an example, take the code below, it’s a distilled summary illustrating the problem, and it took me a fair amount of time to work out why g++ was rejecting it.

template <typename T> class proc {
  struct inner {
    enum class mode { A, B } m;
  };
 
  inner *impl(inner::mode m) { return nullptr; }
 
public:
  void do_stuff() { auto p = impl(inner::mode::A); }
};
 
int main(int argc, char **argv) {
  proc<int> p;
  p.do_stuff();
}

The error according to GCC

martijn@martijn-laptop:~/Software$ g++ file.cpp 
file.cpp:7:9: error: expected ‘;’ at end of member declaration
 inner * impl(inner::mode m) {
         ^~~~
             ;
file.cpp:7:25: error: expected ‘)’ before ‘m’
 inner * impl(inner::mode m) {
             ~           ^~
                         )
file.cpp: In instantiation of ‘void proc<T>::do_stuff() [with T = int]’:
file.cpp:18:12:   required from here
file.cpp:12:6: error: expression cannot be used as a function
 auto p = impl(inner::mode::A);
      ^

The error according to clang

martijn@martijn-laptop:~/Software$ clang++ file.cpp 
file.cpp:7:14: error: missing 'typename' prior to dependent type name 'inner::mode'
inner * impl(inner::mode m) {
             ^~~~~~~~~~~
             typename 
1 error generated.

Short and to the point - but most importantly, it gives a valuable suggestion as to what is wrong in the code. In this case, because the class proc is actually templated, the type of inner::mode depends on what the type of T is (hence it is called a dependent type). What threw me off here is that the struct inner was only kept as an inner struct as an utility in the implementation of the class and in fact, does not depend on T at all. However, as it is an inner class, when it has to assign a symbol to struct inner it will assign it proc<T>::inner, for whatever values T gets instantiated as. This means that from the compilers perspective it has to be a dependent type - and hence the typename is required to resolve the ambiguity that might otherwise arise.

Setting up the Altera Quartus JTAG Programmer on Ubuntu 16.04

Sat, 18 Mar 2017 21:40:19 +0000

Preparation:

Run dmesg -w to show you the logstream of USB devices connecting and disconnecting

Before launching Quartus

Before starting the JTAG programmer in Quartus, the dmesg entries should look like this:

[27810.071135] usb 1-1.2: new high-speed USB device number 18 using ehci-pci
[27810.163900] usb 1-1.2: New USB device found, idVendor=09fb, idProduct=6810
[27810.163905] usb 1-1.2: New USB device strings: Mfr=0, Product=0, SerialNumber=0

Start the Quartus Programmer

Starting the programmer will do 2 things:

Load the JTAG server (jtagd --user-start --config ~/.jtagd.conf)
Load the driver for the USB connection

[28122.900173] usb 1-1.2: new high-speed USB device number 25 using ehci-pci
[28123.010811] usb 1-1.2: New USB device found, idVendor=09fb, idProduct=6010
[28123.010818] usb 1-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[28123.010822] usb 1-1.2: Product: DE-SoC
[28123.010825] usb 1-1.2: Manufacturer: Altera
[28123.010828] usb 1-1.2: SerialNumber: DE-SoC-003-03503

Unable to select the device in the Quartus programmer?

The most likely issue is that currently, you do not have permission to access the device, as by default, USB devices are only accessible as root. Luckily, (as long as you have root access in your system), you can fix this.

Figure out where the USB device lives on the filesystem. In UNIX, everything is a file, including USB devices. Go to /dev/bus/usb, where you will find several folders. The first level indicates the (internal) USB hub, the second level the device number. As seen before, my device has number 25, therefore there is only 1 candidate: /dev/bus/usb/001/025.
Set read-write permissions for all users on that file, eg sudo chmod 666 /dev/bus/usb/001/025. This has to be done every time you plug in your FPGA, which can get very tedious.
Luckily, the linux kernel devs have thought of this and added a feature called udev rules. These are commands that are run, as root, when a USB device of a specific signature connects. Create the file /etc/udev/rules.d/52-usbblaster.rules. Any number between 0 and 100 and name will do but usually the high and low 10 are reserved for system purposes. Do make sure it adheres to the format {number}-{name}.rules. In this file create one udev entry. Mine is shown below. Ensure that the idVendor and idProduct actually match the codes displayed in your dmesg output. The rest of the entry is self-explanatory. Finally, restart the udev service by sudo service udev restart, forcing it to reload the udev rules.
Check whether your udev rules has worked. Replug your USB connection and verify that your device now has read and write permissions for all users, eg:

# The USB Blaster UDEV entry in /etc/udev/rules.d/52-usbblaster.rules
SUBSYSTEM=="usb", ATTR{idVendor}=="09fb", ATTR{idProduct}=="6010", MODE="0666"

$ tree /dev/bus/usb
.
├── 001
│   ├── 001
│   ├── 002
│   ├── 003
│   ├── 005
│   ├── 013
│   └── 025
├── 002
│   ├── 001
│   └── 002
├── 003
│   └── 001
└── 004
    └── 001

# After successfully applying the udev rule or manually setting the permission
ls -lh /dev/bus/usb/001
crw-rw-r-- 1 root root 189,  0 Mar 18 21:22 001
crw-rw-r-- 1 root root 189,  1 Mar 18 21:22 002
crw-rw-r-- 1 root root 189,  2 Mar 18 21:22 003
crw-rw-r-- 1 root root 189,  4 Mar 18 21:22 005
crw-rw-r-- 1 root root 189, 12 Mar 18 21:22 013
crw-rw-rw- 1 root root 189, 35 Mar 18 21:27 036  # all users have read/write permission

Any further issues?

I believe this should be a comprehensive guide for getting Altera FPGAs working on Ubuntu 16.04. If you still have issues, please leave a comment so we can resolve it and add it to the guide.

Context switching on an AVR

Fri, 27 Jan 2017 14:50:19 +0000

Modern operating systems are run on are complicated beasts. They consist of millions of lines of code, providing a wide range of functionality. At the core of these tasks is the ‘kernel’. This core section of the operating system has a single goal: providing a machine-independent abstraction of the hardware. The rest of the operating system and other applications can then be built on top of the kernel to use this interface, meaning that applications should run correctly on every machine type supported by the operating system kernel.

When using a computer, the functionality of the kernel is hidden away as much as possible - it’s merely there to provide services to applications. However, in order to really understand what the kernel does (or could be doing), it can be useful to implement a kernel for yourself.

Kernels are big and complicated so it immediately raises a question: what section of a kernel should we create? The diagram above splits out the hardware into three sections, CPU, memory and devices (eg. keyboard, mouse, display or speakers). Let’s focus on the first one - the processor. From a high level overview processors are simple. You feed them instructions, and those get executed. The tricky bits are in the details. How do you make the CPU load instructions when it isn’t on yet? Now it’s on, do you always run your CPU at full speed, or do you occasionally turn sections off to preserve battery power? What do you do if this specific CPU does not support an instruction used in the program? Lastly, kernels can execute more processses simultaneoulsy than the machine as processors. How do you make (it appear to) do so?

Context switching

The answer to the last question is deceptively simple. Assume we have some set of programs and a single processor that can execute one program. We start it off by telling it to execute the first program in the set. After a few milliseconds, we interrupt the processor and load the second program. Again, a few milliseconds later, we interrupt the processor and load the third program. This would work fine, assuming that programs only run for a few milliseconds - but in reality this is almost never the case. Therefore, after we’ve run every program in the set for a few milliseconds, we can start back at the beginning, giving each some more time before, again, being interrupted. We are essentially dividing up the full program execution of every program into little sections and interleaving those on a single processor, giving the appearance of all of them running simultaneously. However, the user application may have no idea that this is happening and therefore, we need to take care to preserve the state of the program. After each execution section, we store the state of the program to memory. This way, when the program gets its next bit of execution time, we can load the old state and start again where we left off. This makes the process transparent for the running program. As far as it knows, it simply never stopped executing as all values in the program are exactly the same, even though the processor has been executing a different program in the meantime.

This process is known as context switching, in which ‘context’ refers to the state of the program. This state is described by the internal state of the processor, meaning the registers. These contain often-used variables and information about the current execution, eg. what is the next instruction I should be executing. The memory (RAM) used by the process is usually not included as the ‘context’ of the process, but that is because every process already operates on its own bit of memory - which does not cause clashes when swapping out the executing program during a context switch.

Why on AVR?

To be continued with as much operating system as I can fit on an AVR - stay tuned.

Hello again

Thu, 26 Jan 2017 14:50:19 +0000

Demystifying Node.js

Sun, 26 Jul 2015 21:10:32 +0000

Node.js: it’s quick to write, easy to understand and performs very well in applications which are I/O bound for performance. The high level of abstraction offered by Node.js also has a disadvantage. Without looking at the C source code, which is something I cannot imagine many Node.js developers will do, the inner workings can appear to be almost magic. Node.js handles all the nitty-gritty low-level stuff for you: ensuring that the proper callbacks are executed at the proper time with proper values for the variables. This is great and all, but if the server application has to perform actual computation the performance suffers due to the interpreted and dynamic nature of JavaScript. The usual choice is to write the high-performance part a different language which can be compiled to an object file (C/C++) and then add the compiled program as an addon, a process which is documented here.

Why?

As a little side weekend-long side project, I decided to rewrite the underlying code of a small subset of Node.js in C++ without the use of any external libraries. One might wonder: why? Node.js already works perfectly well, but as previously mentioned the high abstraction level distracts from the underlying and supporting mechanisms and one of the goals is to display a simplified version of the inner workings of Node.js. Furthermore, by writing the top-level code in C++ over JavaScript it becomes trivial to perform high-speed computation directly in the server application. Lastly, it’s similar to the early days of Facebooks’ HipHop PHP compiler, which took their entire PHP codebase and generated a single binary server application. This, optimized, single-binary approach obviously has advantages in performance, ease of dependency management and deployment. Lastly, C++ is known for being very complex which comes with the huge amount of control you have and it includes rules which might be hard to grasp. JavaScript is quite the opposite. However, C++ does not have to be hugely complex which is why I’m trying to mimic the easy-to-understand nature of JavaScript in vanilla C++ (that is, without any #defines to hide some syntax).

Restrictions

My own restriction to avoid external libraries and keep the codebase simple has some consequences. It would have been easy to use Boost.ASIO or libuv (which is used internally by Node.js). However, not relying on libraries entails writing a custom asynchronous engine which can be very complicated and hard to understand. Therefore, I decided to use a very much simplified engine, which does not support arbitrary nesting of callback functions. This has some consequences for the designs, but the arising problem: keeping track of state can be solved by using an external structure containing some additional data for each socket connection. Furthermore, low-level concepts such as UNIX socket handling and threading in C++ will also appear in this design.

JavaScript or C++?

TCPServer server ([&](Socket &socket) {

  console.log("Client connected");
  console.log("address: " + socket.remoteAddress());
  console.log("port:    " + socket.remotePort());

  socket.on("data", [&, socket](Data data) mutable {
    //Handle the received data
  });
  socket.on("end", [&, socket](){
    //Handle the client disconnect
  });
});
server.listen(1338);

var server = net.createServer(function(socket){

  console.log("Client connected");
  console.log("address: " + socket.remoteAddress);
  console.log("port:    " + socket.remotePort);

  socket.on("data", function(data){
    //Handle client message
  });
  socket.on("end", function(){
    //Handle client abort
  });
});
server.listen(1338)

Spot the differences!

At first glance the snippet above appears to be JavaScript. It’s easy to write and easy to understand. Almost everything is specified in terms of callback functions. It features the familiar Node.js pattern of `socket.on(“event”, callback). However, it is not JavaScript. It’s C++. The most notable difference between JavaScript and C++ in this snippet is the syntax for functions, which are first-order citizens in JavaScript whereas C++ has to rely on its implementation of lambda functions. The syntax is quite a lot more verbose in C++, but in turn it offers a lot more control!

End-to-end encrypted messaging

Sat, 04 Jul 2015 00:33:42 +0000

Description for an end-to-end encrypted datagram service supporting multicast

Features

low network overhead for the security offered
- ~10 bytes for every group member
- ~250bytes for an encrypted public key for multicast
- ~250bytes for the signature (similar to SSL/TLS)
supports multiple recipients with minimal bandwidth overhead
anonymous message sources, unknown message content (as seen by the managing servers)
signed messages
only the endpoints know the group compositions, none of servers do

Introduction

Node overview

S : Source, the device sending the message
P1 : First proxy (randomly chosen by S)
P2 : Second proxy (responsible for the message division, specified by S to P1)
CS : All user devices have a connection to this node in order to receive messages
T : Target, the destination of the message

Scrambling

Generate a temporary key
Encode the payload with the key
Prepend the key to the payload
Encrypt the key and payload combination with a supplied public key

Setup:

All devices keep a connection with the central server for receiving messages.
All devices not behind a NAT (ie. servers) without bandwidth limits (ie. mobile devices) act as proxies.
Get data from the central server: its public key and a list of proxies with their public keys.
Generate a local key pair for signing.

Operations

Creating a group

generate a key pair locally
add yourself to your local recipient list, which contains the ids and public keys of each member

Adding a user

The user already in the group

prompt for a target id to add to the group
prompt for a passphrase
encode the group key pair and recipient list with the passphrase
send the message to the target id

The user being added

Sees that he has received a message for which none of his private keys work
prompt for a passphrase
if it works, add the recipient list and private key to its cache
for every recipient except itself send scrambled with private_key_G
- the updated recipient list
- the new users public key

Sending a message

Contents of the initial message, which are sent from S to random proxy P1
- The message, signed with the private_key_S, encrypted with public_key_G
- S encrypted with public_key_G
- The recipients, encryped with the public_key_CS
- A specification for which P2 to use (random)
- public_key_G encoded with public_key_P2
P1 forwards the message to P2, contents:
- The message encrypted with public_key_G
- S encrypted with public_key_G
- The recipients, encryped with public_key_CS
- public_key_G encoded with public_key_P2
P2 splits the message into one for every recipient, contents of a single message:
- The message encrypted with public_key_G
- One of the recipients, still encrypted with public_key_CS
P2 scrambles the payload of every message using the groups public key (which gets decoded with the private key of P2)
P2 scrambles the entire message with the public_key_CS
P2 sends every message to CS (possibly through a randomized additional proxy if there is low traffic to avoid traffic pattern detection)
CS descrambles and then decodes the destition of all messages and sends them to the different T
T decodes the message using private_key_G and descrambles (step 4), decodes again (step 1) and obtains the cleartext.
T verifies whether the signature is correct and passes on the message to the receiving application

Who knows what?

S

Message contents
All recipients
Message signature
Choice of P1 and P2

P1

S
Choice of P1 and P2

P2

Choice of P1 and P2
Group public key

CS

Choice of P2
A single recipient

T

Message contents
Message origin
Message signature

Possible problems:

Source bandwidth increases with group size (if every group member would use their own key pair for encryption)
- Fixed by adding the shared group key pair
Keeping participants anonymous whilst verifying their authenticity
- Fixed by the requirement for a passphrase when adding people (which has to be distributed over another secure channel)
Sending data to anonymous receivers
- No need for anonymous receivers due unknown message contents and anonymous transmitters
Latency due to a triple proxy (P1, P2, CS)
- can be shortened to a double proxy by sacrificing multicast possibilities (eliminating P2)
- But the double proxy still ensures anonymity
If you have control over the paths between P1, P2 and CS you can trace the path of the packet.
- Their input and output content at P2 is entirely scrambled and combined with a high traffic level and a short, randomized wait, this prevents traffic flow analysis.
If you have control over all servers you can trace the path of the packet
- Indeed. As long as not all the servers get hijacked the system is still anonymous though and due to the random choice of P1 and P2.
- The content of the packet will always remain unknown to the hijackers due to the end-to-end encryption.

FPGAs?

Tue, 26 May 2015 14:54:32 +0000

VHDL: hardware design could be more streamlined!

This is a VHDL (VHSIC Hardware Description Language (Very High Speed Integrated Circuit) - so much for concise abbreviations) snippet which synthesizes to a module which simply passes on 4 wires to some external LED output, synchronized to a central clock.

library ieee;
use ieee.std_logic_1164.all;
use ieee.numeric_std.all;

ENTITY io_led IS
  PORT(
    clock     : in std_logic;
    reset     : in std_logic;
    leds_status : in std_logic_vector(3 downto 0);
    leds_output : out std_logic_vector(3 downto 0)
  );
END io_led;

ARCHITECTURE behaviour OF io_led IS

BEGIN
  PROCESS(clock)
  BEGIN
    IF rising_edge(clock) THEN
      leds_output <= leds_status;
    END IF;
  END PROCESS;
END;

The missing delete

Mon, 25 May 2015 23:25:23 +0000

void ClipShareRunner::readFromSocket(QString str) {
updatingClipboard = true;
QMimeData * mimeData = new QMimeData();
/*
Process QString str and add to the information to the QMimeData object.
*/
QApplication::clipboard()->setMimeData(mimeData);
updatingClipboard = false;
}

Any C/C++ programmer who glances over this piece of code will figure out quite quickly that it has some unintended consequences: a memory leak caused by mimeData = new QMimeData() without ever calling a corresponding delete mimeData;. That’s the general pattern for allocating persistent memory on the heap: new and delete. Modern C++ is very much fond of its RAII principle (Resource Acquisition Is Initialization), an obscure term for a simple principle. Put your news in the constructor, put your deletes in your destructor and you are almost guaranteed to never ever have a resource leak.

However, in this case you are not just passing the pointer for the new content of the clipboard, you are also (implicitly) passing on the ownership of the data the pointer points to. Therefore, it is not our responsibility any more to clean up after ourselves. If you attempt to and the object has already been moved, you are greeted by a familiar opponent: the double free error.

We’re only left with new. The corresponding delete is nowhere to be found in our code. Its instruction will be buried somewhere deep inside some compiled library binary, never to be seen again by people: just a few bytes in a sea of apparent pseudo-randomness, making sure that your computer does not crash if you have placed things on the clipboard too many times. Thank you bytes.

In retrospect, the reasoning behind also implicitly passing the ownership of the data and thereby the responsibility to delete it is completely sound: Only the operating system is able to keep track of whenever the clipboard and thereby its mimeData object have been reassigned. In current machines with huge surplusses of memory reallocating is not that efficient anymore: deleting the old object and just allocating a new one. Therefore there is no way to know for the application where the clipboard object has gone and as such it will not be able to clean up after itself.