Another important change is that ifnet now supports openrc style. As bash arrays are not allowed by openrc, ifnet caused Bug #369711. After networkmanager-0.9.2.0-r2, ifnet can read both the old baselayout style (with bash arrays) and openrc style (without bash arrays), but it only writes to /etc/conf.d/net with openrc style.

I have a list/queue of 200 commands that I need to run in a shell on a Linux server. I only want to have a maximum of 10 processes running (from the queue) at once. Some processes will take a few seconds to complete, other processes will take much longer. When a process finishes I want the next command to be “popped” from the queue and executed.

Solution: write a makefile with the following content:

all: usera userb userc....
usera:
imapsync usera
userb:
imapsync userb
....

Then just run

make -j 10 -f makefile

I think the answer is really brilliant as it not only makes use of available tools that are quite mature, but also solves the problem fast with little effort. If the situation is getting more complicated, the solution can still be easily adjusted. xargs is also a good solution btw.

So far, libbash can generate correct ebuild metadata for 8414 out of 27586 ebuilds. What is blocking us from moving on? We need to be able to parse all the eclasses. But there are still a few problems in the parser grammar. On the other hand, our runtime is not complete. In one word, more time is required. This is the ohloh page for libbash where you can learn some statistics of the code.

I wrote an ebuild for libbash and put it in my overlay. Now you can play with it by

layman -a qiaomuf
emerge libbash

Although libbash is not powerful enough to generate all ebuild metadata, it can do something useful now. As I said before, you can always check what libbash can support from these test scripts. One thing you should notice is that we do not support running external command.

The library is quite easy to use. To give you a taste, here is a code snippet showing how to use it:

#include <algorithm>
#include <iostream>
#include <iterator>

#include <libbash/libbash.h>

int main(int argc, char** argv)
{
  // store variable values and function names
  std::unordered_map<std::string, std::vector<std::string>> variables;
  std::vector<std::string> functions;

  // interpret the script specified by argv[1]
  libbash::interpret(argv[1], variables, functions);

  // do what ever you want with variables and functions
  std::copy(functions.begin(),
            functions.end(),
            std::ostream_iterator<std::string>(std::cout, " "));
  std::cout << std::endl;
}

To get it compiled, run

g++ -std=c++0x $(pkg-config --cflags --libs libbash) test.cpp

The program will interpret the script like bash and store the result in the variable map and function name list. All the variables are treated as arrays. For more information, you can check the API documentation.
(P.S.: The library is not stable so please don’t use it in any serious place)

So what’s next? I think it would be improving the parser grammar, figuring out problems from the output of instruo and implement more runtime. People are welcome to make contributions. I’ll continue making contributions after GSoC.

Last but not least, I’d like to thank Google and Gentoo community for giving me such great experience. Thank Petteri for mentoring me. I enjoyed a wonderful summer with you and learned a lot including C++0x, boost libraries, ANTLR, autotools, Scrum and so on. Thank Donnie, Robin and other people who helped me.

OK, I’m going to get myself prepared and seek a job

Now 8414 ebuilds can be handled. The number is increasing again and it’s not that hard to make progress this time because we have fixed the big issues in the parser grammar. However, running instruo is not that fast any more. It is because we can parse more eclasses so there are more things to handle.

I did more than expected in the last week. Here is the list of what I have done in the last week:

• Fix code so that our instruo won’t crash anymore(turns out to be a problem related to thread-safety)
• Get libtool.eclass fully parsed
• Get perl-module.eclass fully parsed
• Get distutils.eclass fully parsed
• Get java-utils-2.eclass fully parsed
• Get kde4-base.eclass fully parsed
• Get python.eclass parsed with one exception: double quoted arithmetic expansion

The above eclasses included all the tasks for grammar improvements that I mentioned in the last weekly report.

This week I will:

• Improve our build system to install something useful
• Improve our ebuild so that the library can be installed and used properly
• Review all the documentation
• Review all the code

What I have done in the past week:

• Completed the work on the new parser and incorporated it to our project
• Improved parameter expansion parsing and its runtime
• Improved built-in and keyword test
• Improved the runtime for case statement
• Improved arithmetic expansion
• Improved the local built-in
• Fixed some minor problems in compound statement and parameter expansion
• Reimplemented the export built-in
• Removed several tokens to avoid conflicts
• Improved here document and here string
• Fixed single quoted string in command substitution

In the following week, I will:

• Get backtracking removal pushed
• Fix our instruo implementation (now it crashes with the new grammar)
• Improve process substitution
• Support redirection without any command
• Reimplement the local built-in
• Fix some minor problems in variable expansion and bash test
• Remove some composite tokens

I started working on this project early and I will start seeking a job soon (I’ll soon graduate from my university). So Petteri and I agreed to end the GSoC on 08.06. As a result, this is the last iteration of this year’s GSoC. I’ll write one more regular report and a final report before the end. I’ll continue my work on this project as soon as I get a job .

My plan for the mid-term evaluation is to be able to parse the ebuilds that don’t inherit any eclass. With a few exceptions, I achieved that more than a month ago. If you read my weekly reports, you might have noticed that we could parse about 7000+ ebuilds at some point. That’s when I achieved the goal.

If you want to see what libbash can handle so far, there are several test scripts where you can find the answer.

But why are there some exceptions? The answer is due to the limitation of parser grammar. The parser grammar was supposed to be working at the end of GSoC 2010. However we found there were still many issues in the grammar while implementing the runtime of libbash. The parser grammar is quite important and I can’t work on the runtime if the parser grammar does not work properly.

Then what are you doing these days? I spent most of the time fixing the problems in the parser grammar. In my GSoC application, I said if I had enough time, I would try to disable global backtracking for the parser grammar. But now this seems to be required. I started working on it from last week and it goes pretty well so far.

Some people doubt if I could completely disable backtracking. I’m not a compiler expert and I don’t have the confidence to say if bash is LL(k). As far as I see, I can handle most of the syntax with LL(k) parser. But what I am really trying to do is to remove the global backtracking option of ANTLR grammar. With that option, ANTLR will automatically backtrack so there might be many places where you could have avoided backtracking. It also makes the problems in the parser grammar even more complicated. It’s highly recommended not to use that option in production code. As a result, I’m trying to use left factoring, syntactic predicate and local backtracking option instead of the global one. So far the parser grammar is faster and it’s much easier to fix the problems in it. I’ll give the performance comparison in the next weekly report.

What I have done in the last week:

• worked on a new parser grammar without the global backtracking option

Plan for this week:

• finish the work with the new parser grammar and improve unit tests
• incorporate the new parser grammar
• fully support here document
• improve the rules for parameter expansion
• improve the rules for built-in and keyword test
• fix the problem with \newline

These are what I have done in the past week:

• Used bash to verify unit tests
• Supported thread safety
• Turned off backtracking for the command rules
• Left factored the rules for pipeline and function definition
• Improved double quoted string handling
• Worked on upgrading to ANTLR 3.4
• Started working on removing global backtracking

Note that we support thread safety now and our version of instruo can finish its work in less than 1 minute. We still haven’t upgraded to ANTLR 3.4 because of some upstream bugs. The process might take longer.

In the next few weeks, I will mainly focus on one story: removing global backtracking. I hope it can be done in two weeks.

• Finished setting up the CI server
• Broke down the walker grammar to reduce compile time
• Unified header protection coding style
• Cleaned up doxygen warnings

two stories are blocked:

• Upgrade to ANTLR 3.4

I fixed some problems in the libraries and reported some bugs upstream. Now I’m still blocked by some upstream bugs. This story is quite important because many stories need semantic predicate. We hope ANTLR 3.4 could support it when backtracking is turned on.

• Make the library thread safe

There are still some problems with the thread safe code, I will continue working on it in this week.

So this week I will continue working on the blocked stories. We will decide what to do next after upgrading to ANTLR 3.4.

• Supported braces in command arguments
• Improved comment handling
• Supported ANSI C Quoting
• Supported shortcut capability for && and || in arithmetic expansion
• Supported arithmetic expression
• Supported break built-in
• Improved our build system to reduce dependencies
• Made arithmetic expansion follow POSIX
• Improved exception hierarchy
• Implemented shift built-in
• Improved the ast_printer utility

A few stories are blocked due to the backtracking problem.

This week I will:

• Upgrade to ANTLR 3.4 and see if it helps solve the backtracking problem
• Fix errors in the CI Server
• Break down walker grammar to reduce compile time
• Use bash to verify test scripts
• Support alias
• Handle options to the local built-in
• Upgrade to Paludis 0.64.1
• Support thread-safety
• Clean up warnings from doxygen

• Supported bash redirection for all kinds of commands
• Supported the special parameter $-
• Supported parsing -o and -a operators for built-in test
• Supported brace expansion
• Implemented eclass parse failure cache
• Supported backslash escapes inside double quotes
• Fixed variable indirection in arithmetic expressions
• Supported regex match operator for keyword test
• Tried to parse here document and improve variable expansion
• Improved CI server configuration

This week I will:

• Support braces in command arguments
• Improve comment handling
• Handle single quoted string in variable reference like $’string’
• Support shortcut capability for && and || in arithmetic expression
• Support arithmetic expression
• Support break built-in
• Support read-only built-in
• Improve our build system to reduce dependencies
• Make arithmetic expansion follow POSIX
• Improve exception hierarchy
• Implement shift built-in
• Try boost::spirit::qi to implement a simple lexer for ANTLR

At the end of this weekly report, I’d like to mention a small tip for bash arithmetic. As you probably have known, you can’t write $(( expression )) as a bash command. But sometimes you just need to evaluate the expression and nothing else. You can certainly call the let built-in but that requires you quote your expression to avoid word splitting. Then some people invent this:

: $(( expression ))

‘:’ is a bash built-in that does nothing. So the argument gets evaluated first and then ‘:’ gets called. I really feel it unnecessary as you can always use bash arithmetic expression:

(( expression ))

This is exactly equivalent to

let "expression"