PEPPr – Python Enhanced PreProcessor

Minor Snags, Now Solved

2018-02-23T16:00:00+00:00

Things have been quiet for a bit in the devblog world and I ran into a few issues that I thought bore mentioning, though none deserve a blog post in their own right. Thus, this.

Turns out, Ply eats syntax error exceptions

Pepper makes use of the Python PLY package, a python reimplementation of the lex and yacc language parsing tools. It allows you to specify the specification for tokens which are then used in a parser grammar that you specify.

Part of this grammar is an error state, which you hit if you input something to the parser that doesn’t match its syntax–your standard syntax error, really.

PLY requires that you have this rule and that it in some way handles this state–it can do anything it wants, really, but the sensible actions are limited to something like “halt the program” or “ignore the error” or “emit an exception”. I chose the last route.

Seeing as Python already had a SyntaxError exception that I could freely raise, I made the error rule puke a SyntaxError with the current parse object.

I also didn’t test the emission of that exception. Which was arguably my first mistake, since it would have caught the next.

Some time later, I’m making it so we can have macros with arguments and I’m chasing a bug where output is just missing from the emitted intermediate file, which in theory isn’t possible.

The way the preprocessor is structured is that it preprocesses input line-by-line and appends the output to a gigantic list (because I have this vauge notion that it’ll be useful to have lying around) before writing the whole thing out to file. With this structure it wasn’t possible to have a partial line being written out–either you succeeded in parsing the whole line, and then the whole file, and wrote it all out, or you broke somewhere in your single line and crashed the whole program with an exception.

What I was seeing was something like:

int main() {
    cout << "some stuff" << endl;
    cout <<
}

as output from the preprocessor. What I was expecting was something like:

int main() {
    cout << "some stuff" << endl;
    cout << someMacroExpansion(a, b c) << endl;
}

Clearly something had gone horribly wrong.

The long and short of it is that PLY was internally eating the SyntaxError my error rule emitted, since the parser was getting something it didn’t know how to handle, and it simply returned the truncated parse tree which happily emitted as much output as it could, which then got written to python.

So much for “errors should never pass silently”, eh?

In the end I made my own exception, PepperSyntaxError, derived from Exception, to emit instead of the default SyntaxError. I’m a little curious what PLY is doing under the hood where eating SyntaxErrors is normal behavior, but I’m not sure I’m motivated to find out.

This ended up feeding nicely into a structure in the preprocessor that catches any internal errors (e.g., ones not caused by bad syntax) and prepends a “Hey, please report this error” with a link to the issue tracker, in the hopes of being proactive in solving user problems when they inevitably hit a corner case I haven’t thought of.

Flirting with Parsing C++

Parsing C++ is one of my worst nightmares.

Somewhat recently I happened across a blog post by a fellow compiler fiddler (indeed, someone who built a full on c parser in python to do lexical analysis!) which, initially, got my hopes up about solving a problem I was having; specifically, that I needed to parse list-like-structures from within parentheses to handle function-like macro expansions, since commas can be meaningful in c++ in a bunch of other places and I initially thought I’d have to parse the arguments themselves.

As Eli of TheGreenPlace notes, parsing C++ is phenomenally difficult in general and explicitly impossible with a YACC style parser. I don’t pretend to understand why or how, I just know I’d like to avoid that complexity as much as I can since it doesn’t seem necessary for the work I’m doing and would add phenomenal complexity–and thus speed cost, in both performance and development–that just isn’t needed.

My “reference implementation” of the preprocessor (read: g++’s preprocessor) appears to handle commas rather roughly as well, which may save me. With input like:

# define TypedThingyMajig(type) std::Vector<type>

int main() {
   auto c = TypedThingyMajig(pair<int, float>);
}

you get error output like:

test.cpp:4:46: error: macro "TypedThingyMajig" passed 2 arguments, but takes just 1
    auto c = TypedThingyMajig(pair<int, float>);
                                              ^
test.cpp: In function ‘int main()’:
test.cpp:4:13: error: ‘TypedThingyMajig’ was not declared in this scope
    auto c = TypedThingyMajig(pair<int, float>);

So yay, I can be heavy handed with the parsing of c++-like structures and save myself some headache.

Minor performance concerns

Pepper is primarily a command line tool. As such, many of the tests I’ve written are assertions against the output written by Pepper when called as a subprocess. This is slow as hell for a python unit testing framework that’s expecting to call a ton of library functions, and is remarkably frustrating for me because it’s slow and clunky as hell.

At some point I intend to sit down and rewrite the tests so they make use of as many module-level function hooks as they can to test equivalent functionality, but as it stands most of the tests are command line calls on carefully structured input files.

While I was developing the function like macro expansions I ran into several instances of some test cases failing intermittently by whacking into the 2-second timeout I allot for subprocess runs, which seemed like a decently high ceiling for the small test cases I was running.

Doing some profiling with SnakeViz and cprofile it looks like most of the program’s time is spend standing up all the individual modules of Pepper–85% of the program’s time is spent handling import statements and the machinery involved in standing up modules. Obviously this is skewed by the fact that we’re using really, really small test cases and aren’t actually doing that much heavy lifting within the preprocessor itself. We’ll have to construct some larger cases to get a decent look into performance before we start fiddling, but it’s something to be aware of as our test case library swells. Nobody likes tests that take forever to run.

Next steps in development

Handling function like macros was one of the last major components that needed to be finished before we could begin more parallel development. Now that it’s done I’m hoping we can start running multiple branches in tandem, and expanding the integration test library.

Next on the docket is handling #if directives, which will involve some parsing of code, which I am very not excited about. We’ll also be building in #pragma functionality and things like __VA__ARGS__ within macros themselves, a prerequisite for a lot of the macros in the Empirical library, which we’ll be using as most of our acceptance tests once we reach a more feature-complete state.

The general roadmap for development is things that block us from compiling a Hello World style program, which really means being able to preprocess a decent amount of the C++ and C standard libraries.

Building Integration Tests

2018-01-30T12:50:00+00:00

written by Cyndy Ishida

Building Integration Tests is my first notable contribution to the Pepper Project. I’m hoping that the tests I design will later be the way we actually pipeline Pepper into a normal compiler. Currently the chain of steps is to run a command like:

Pepper some_file.cpp --output_file some_file.ii

Here it invokes Pepper and runs the output of the preprocessor to the same file name with the extension of ‘ii’ which is just an extension that signals to the compiler to skip the pre-processing stage.

Only in the GNU compiler, you can actually avoid changing the file name with using the ‘-fpreprocessed’ flag.

Based on a global variable ‘CXX_FLAG’ which I’m hoping in the future to read the environment variable for it when it’s being ran for either G++ or Clang. I don’t see there being a huge need to support MSVC compilers, but what do I know.

CXX_FLAG -Wall -std=c++17 -o output some_file.ii

Currently I run that to build the pepper-ed executable and also run the original c++ file to the normal compilation stage and compare both standard outputs, as per Jake’s suggestion.

###Concerns for Pepper with Compilation I would like to actually run Pepper in production and redirect the output from the compiler to the Pepper interface. This would be pretty simple with the subprocess module and storing the information from standard out and error. A few things that trouble me about my current stream of execution moving to production is portability. I don’t really see spawning a ton of threads as a viable way of pipeline-ing Pepper, especially if this would ever be used in a memory and processor(s) intensive situation. I think it would be worth looking into.

###GPU Support As stated above, I really only thought about using Pepper with clang or g++ but applying NVICC support would be cool. Well maybe just, modularized enough to support any C++ compiler.

For Want of Another Parser

2018-01-11T16:00:00+00:00

written by Jake Fenton

Here’s my current problem: Macro expansion

Suppose you have a macro:

#define quail(a, b) a + b

Suppose you have another macro:

#define potato(a, b, c) quail(a, b) + c

And then you have code:

int something = potato(1, 2, 3);

If you’re the preprocessor, you’ll have some kind of parser that scans the incoming code for identifiers to expand. You see potato and go “Oh, hey, I have a potato macro defined in my symbol table! I should expand that.” And the parser pulls the arguments for potato, builds the syntax tree, builds some magic and then outputs int something = quail(a, b) + c.

Which is great! We expanded the potato macro….but now we have the quail macro to also expand. The preprocessor spec is aware of this, and has the following to say:

All arguments to a macro are completely macro-expanded before they are substituted into the macro body. After substitution, the complete text is scanned again for macros to expand, including the arguments. This rule may seem strange, but it is carefully designed so you need not worry about whether any function call is actually a macro invocation.

This is a relatively straightforward rule: If you expand a macro, you should parse the expanded line to see if there are any other macros in the expansion. Unfortunately this introduces some complexity in the lexer and parser themselves.

The PLY lex/parse stack takes an input stream and tokenizes it, according to the language specification in the lexer. This is an on-line (i.e., streamable) but single-pass process. The token stream, once ended, then gets passed to the parser which determines if it can build a valid parse tree. (This process requires the incoming token stream to be finished.) Once the parse tree is built, we execute the Preprocess() function call on the root node, which begins building the output from the preprocessor which, in theory, can be handled by just the C/C++ compiler.

Aside: the Abstract Syntax Tree

This should probably be a post in and of itself, but it’s necessary for full understanding of the issue:

The Abstract Syntax Tree (AST) is the structure that holds sets of tokens, as specified by the language specification (lexer) and the grammar specification (parser). The tree holds the entire structure of a program–the root node is typically a Program node, which contains many Statement nodes. The Statement nodes can derive to things like control statements, arithmetic operations, etc.

From Wikipedia

The tree is abstract in that the ‘arguments’ to each of the nodes are variable, so long as they fit the format of the node. Thus, you can have abstract ‘statements’ within a ‘for’ or ‘while’ type of node.

To build the output from this tree, be it intermediate code for a compiler or c++ code without preprocessor directives for a compiler, you’d call some method on the root node that would cascade down the tree, building the output from each of the nodes and stitching it together to fit the language.

…back to the problem…

So, the issue we have is that when you expand a macro you’re substituting C++ code–maybe with some arguments–into an existing token stream. This changes the stream significantly, but there isn’t a way to re-lex and re-parse it. We’re in the parsing stage, which means lexing is complete. Re-lexing the string could result in input that cannot be parsed, if it doesn’t use the grammar correctly. If we can’t re-lex and re-parse the string, how do we check for additional macros to expand?

Some possible solutions:

Parse all macros for identifiers of other macros–if found, auto-expand.
- Problem: macros-within-macros could have argument lists that would need to be parsed and expanded. Difficult to do with variables.
Instantiate a second parser/lexer stack to chew through ‘lines’ of preprocessor output, within the preprocessor, to check for additional macros to expand.
- Good: relatively lightweight, allows preprocessing to be a contiguous process (e.g. no multi-pass), which should have performance benefits
- Bad: requires building another ‘preprocessor’ stack to handle finding identifiers and processing argument lists for expansion; more code to maintain
Re-parse the entirety of the output from the preprocessor to check for more macros to expand
- Good: Simple-ish to implement, less repeated code
- Bad: Have to handle including the macro definitions for the re-run instances, inefficient as all hell, gonna be ridiculously slow.

The plan

I’m going to build a simplified parser that will scan for identifiers in a line and work to expand macros. It’ll inherit a symbol table of macro expansions from the root parser. Hopefully this will bypass the issues with having to “re-parse” a line, since we’ll be using an external parser, and won’t give too much trouble with expanding macros.

This method should be more efficient than others, since we’ll only be re-scanning individual lines that have had a macro expanded in them, and not the entire program.

Caveat: Before I build an entirely new (though simple) parser, I’ll try some experiments to see if we can get away with making a second instance of the existing parser.

Best Practices

2017-11-14T16:00:00+00:00

written by Jake Fenton

In my first programming job, as a freshman at Michigan State, I was told by the lead engineer of the project (Michael Crusoe) that, for best practices, you should write policy and then write software to enforce that policy. That’s been my goal with how I’ve set up best practices for the Pepper project.

There’s really three things, I think, that are important in a library, from a development perspective. It should have automated tests, it should be readable code and it should have documentation. All of this is important in general but it’s absolutely critical if a project is to be open-source and to have outside contributors.

Here’s a quick overview of how we use these within Pepper:

Testing

I was lucky enough to be introduced to automated testing very early in on in my development career. The khmer project, which you’ll note is where a lot of these insights come from, made extensive use of automated tests. They had about 80% coverage when I joined the project and above 90% when I left. (Very little of which actually had to do with me, honestly.)

The tests were fantastic peace of mind–you could run them after you added new code to an existing script to make sure the rest of the script ran as expected. You could run the tests after a major refactor to make sure everything worked still. Dependency update? New release? Pull request from an unknown, outside contributor? You can run the tests and make sure nothing is broken.

The Pepper project has an automates test framework in place, pytest, that can be run locally and by the CI server. If any test fails, the CI will fail your branch and you won’t be able to merge it in. (Thanks to Github for adding that policy.)

Also like the khmer project, Pepper enforces test coverage. If you add code that isn’t covered by the test suite, the CI will fail your build. This ensures that if you add new functionality you have to add the corresponding tests pass well. The coverage check is run against the diff of what you add, however, so you won’t get yelled at for existing holes in coverage. (Though, in theory, if we start fully covered and we require that everything added is fully covered, no coverage holes can happen.)

Of course, you could just write a test that passes always, no matter what your code does. We don’t have a way to automate enforcement of writing good tests–that will still require human eyes. Code review is still also important.

With the automated enforcement of both passing tests and test coverage Pepper should remain a well tested library, covered by easy-to-use automated tests which are enforced by CI and can be run locally.

Readable Code

Most will agree that having a consistent coding style (naming scheme, tab format, brace format, etc) across a project is important, especially when accepting things from outside contributors–typically people disagree on what that style ought to be, but that’s beside the point.

Pepper is no exception to this. Pepper uses the Flake8 linter to run pep8, pyflakes and prevent circular dependencies. This, as with testing, is enforced by CI, just as adding tests is–and the lint check is run against the diff, so you won’t get yelled at for style problems that someone else introduced. (Though, again, if we start with none and we don’t allow any in merges then there shouldn’t ever be any.)

Enforcing style like this should prevent unreadable, confusing code down the line, keeping the project’s barrier for entry relatively low.

Documentation

Currently we don’t have automated enforcement for documentation coverage, but there do appear to be packages that will allow for enforcement of that. Currently we have documentation for the master branch being built by ReadTheDocs, though GitHub doesn’t view ReadTheDocs as a CI server, which is unfortunate. As it stands though, there’s a neat little badge on the readme that shows the status of the documentation builder.

Some parts of the documentation are also auto-generated from the docstrings of the source code, making it easy to directly document code and parameters.

Having documentation easily accessible, both for the contributors and for people looking to use the library, is important to keeping the barrier to entry low. This is part of why I maintain this blog alongside the official documentation.

Code Review

Thought the use of Github’s ‘protected branch’ system, the Pepper Master branch is restricted, requiring an ‘approved administrator’ to review code in a pull request before allowing that code to be merged into the codebase. There’s also a contributor checklist that contributors are supposed to copy into the comments of a pull request and check off as they complete the parts. This makes sure that contributors go through the steps they’re supposed to for their contributions (making sure it’s mergable, making sure it’s covered by tests, etc) and that the administrators and maintainers of the project make sure people followed the guidelines.

Requiring code review is also a good way to make sure that any written tests are written well, that added documentation makes sense and that any new code actually does what it’s supposed to.

Next Steps

Now that theres automated enforcement of best practices in place, development can begin in earnest. The project is probably still too immature to attract many outside contributors, but my hope is that with all this infrastructure in place it will be easy for them to join and for the existing team (i.e., me) to trust the contributions they make.

Designing the Token Specification

2017-09-22T16:00:00+00:00

written by Jake Fenton

In this post I shall document the saga of my developing a token structure for the preprocessor.

I’ll be heavily referencing the GNU C Preprocessor specification–specifically section 1.3 on tokenization–throughout this post (which I have an offline copy of thanks to wget).

Wait, what’s a token?

Ah yes, this is supposed to be a cold-start guide to writing a preprocessor.

(Again, disclaimer that some of this is possibly wrong–While I did take a compiler course I was on the ‘student’ side of the interaction, and I do not claim to be an expert on compilers.)

In the general sense, a compiler is a system for translating a language. Languages have an alphabet from which symbols can be arranged in ways that have meaning. Tokenization is the process of taking a stream of these symbols, grouping them and classifying them. Tokenization acts only on the symbols–it doesn’t determine grammar, necessarily.

Take this basic example of a math equation:

y = 5 x + 17

We might tokenize thusly, where [] denotes a token:

[variable, y] = [number, 5] [variable, x] [operator, +] [number, 17]

This stream of tokens will later be fed to a parser which is responsible for building the abstract syntax tree which is what will actually describe how these tokens interact with each other.

But that’s a topic for later–right now we just need to break apart some c++ code into tokens, which means we get to learn regex!

Wait, what’s a regular expression?

Regular expressions are hands-down my favorite part of my compilers course.

A regular expression is a way of describing a pattern that would get used in something like grep or the find-and-replace dialogue. I think the best way to learn them is by seeing them and then experimenting. I highly recommend the site regexr as a resource for learning.

We’ll use regular expressions to define what our tokens can look like. Within our lexer (the thing responsible for building the token stream from the input stream of the language) regular expressions will be transformed into a set of finite state machines, which will be used to figure out what the input could match.

Anyway, back to tokens.

(chunks of this will just be me regurgitating the aforementioned GNU page)

According to the GNU reference pages, from the preprocessor’s view there are five categories of tokens we care about:

identifiers
preprocessing numbers
string literals
punctuators
other

An identifier is what you would expect–anything that could also be an identifier in C/C++; starting with an underscore or letter, and then alphanumeric with underscores throughout. The regex for this would look something like:

[_a-zA-Z][_a-zA-Z0-9]+

A preprocessing number includes the normal integer and floating point numbers that you can use in C but also includes a weird format of numbers that will make for a very ugly regex; the formal specification requires an “optional period, a required decimal digit, and then continue with any sequence of letters, digits, underscores, periods, and exponents.”

“Well that’s not so bad,” you say. “What’s so hard about that?”

The exponent can be any of ‘e+’, ‘e-’, ‘E+’, ‘E-’, ‘p+’, ‘p-’, ‘P+’, or ‘P-‘.

So, I’m going to attempt to sketch out an example regex of what that might look like:

\.?[0-9]([0-9]|(e\+)|(e\-)|(E\+)|(E\-)|(p\+)|(p\-)|(P\+)|(P\-)|[a-zA-Z])*

Note that the character class [a-zA-Z] has to come AFTER the definition of all of the exponent groups, or the regex will count the leading character of the group as a letter and not an exponent.

According to the magical GNU document, this weird definition is to “isolate the preprocessor from the full complexity of numeric constants.” Initially, I thought this was ridiculous, as we would have to somehow determine if the number was valid or not before using it in operations, and then I remembered that it’s not our job to do that, as we just build tokens, occasionally expand things, and pass the lot of tokens onto the compiler.

(This is something I’ve been quite bad about, actually, is remembering how limited our responsibilities are. I’ll have to work on being better at that.)

String literals are pretty much what you’d expect. One thing to note is that arguments to #include are technically string literals.

Quoting from the magical GNU document again, “There is no limit on the length of a character constant, but the value of a character constant that contains more than one character is implementation-defined.” Again, though, it’s probable that we won’t have to concern ourselves with that as it will fall to the compiler. That said, the fact that the GNU document notes this is suspicious, as the document is exclusive to the preprocessor.

My sketch of a regex for string literals is:

('|")((\\['"tn])|[^'"\\])*\1

Punctuators are the standard set of puncutation meaingful to C and C++, which I looked up in the C++ specification doc I have:

If we ignore the keywords that C++ cares about (bitor, etc), which will be read as identifiers to the preprocessor, and the concatenations of other operators, and digraphs and trigraphs, it’s a pretty normal set of punctuation.

And of course, my regex sketch: [{}:;,?%&*<>=/!]|[\[\]\(\)\.\^\-\|\+]

Next Steps

So, now that all the tokens are skethed out we’ll just need to define some test cases for the broad token types and see if we can correctly parse them. If/when that happens we’ll break the tokens down into what the preprocessor actually cares about, and then attempt to act on them.

Once we can handle lexing the incoming files we can work on the actually hard problems like passing the tokens to the compiler, including files and other fun stuff.

Initial research

2017-09-22T16:00:00+00:00

written by Jake Fenton

Before beginning this project I’ve been assembling some prior work to try and get a handle on what this project will entail. To that end, I initially searched for existing C/C++ preprocessors written in Python. While I found several projects that somehow combined python and C/C++ preprocessing they were, for the most part, somehow inserting C/C++ preprocessor stuff into python and not actually a preprocessor written in python.

I eventually happened upon CPIP, which was indeed a C/C++ preprocessor that was written in python, but unfortunately was last updated in 2012 and the mailing list and developer email are sadly silent. We considered using CPIP as a starting point for the project but CPIP unfortunately had a homebrew (and thus unmaintained) lexer/parser system. I made the decision to implement the preprocessor from scratch and make use of CPIP as an architectural reference.

To that end, I’m currently making friends with the GNU C Preprocessor page (which I made sure to download an offline copy of) and will maintain this blog as I work on this so the next poor soul to attempt a project like this will have somewhere to start.

(Writing this will also hopefully help me to organize and document my thoughts so I don’t cowboy code my way through this whole thing.)

A quick primer on preprocessing

Disclaimer: this is probably fantastically wrong in some way that I’ll discover down the line in an explosive fasion, but until then….

Preprocessing is the stage before actual compilation of the program occurs, as the preprocessor works to resolve the #include statements (and any #defines, #ifdefs, macros, etc). When the preprocessor is finished you could dump a file that would be perfectly syntactically correct C++ that could get fed to a compiler that knows nothing about preprocessor language or macros. It (very probably) would be a very large file, since all the files #included in would be just copy-pasted in (and all the files that file #included, and all the files those files #included…)

So, thankfully, this means the concerns of this project don’t extend to the full length of the 1368 page C++ specifications document I’m working from the isocpp page, and for the most part confines us to the sections on lexical conventions (p. 17-24) and preprocessor directives (p. 426-437).

Also, thank you to the designers of C++ for providing a wonderful lexical tree all ready to go on page 426.

That’ll be useful when I’m hacking together my Ply and Lexx setup.

Anyway, back to the primer:

There are a couple things complicating our journey to build a preprocessor. First, most preprocessors don’t just construct a massive C++ file that gets handed off to the compiler. If we did, the errors emitted by the compiler would be impossible to trace back to your own code. Most existing preprocessors pass the tokens they extract from the source files to the compiler directly via a binary stream. Since we seek to replace the functionality of existing C++ preprocessors, we’ll have to do the same.

There’s also the new functionality that we’re going to build out, specifically the ability to include snippets of python in preprocessor directives that will be executed at compile time with a shared namespace to access the symbol table used by the standard preprocessor macros. These python directives will be capable of emitting output that will be copied into the program.

Something important to note is that we have no intention of making it possible to define a macro in python that will be executed during runtime–that kind of cross-language weirdness is out of scope. We’re just looking to crank up the power of C++ macros, basically.

So, in summary:

Fetch and copy-paste any #included files
Handle the logical/flow control preprocessor directives a. Include/remove code as dictated (#ifdef and the like)
Expand macros
Remove comments
Pass the token stream, with diagnostic/location metadata, to compiler

I’ll start with sketching out a basic lexer setup and then expand it with tests.