Posts Tagged GSoC

Current Status of Libbash

Glad to see that libbash is accepted by Google Summer of Code this year. Congratulations to André Aparício! Petteri and I will both mentor this project during the summer. Before GSoC begins, I’d like to explain the current status of this project.

I worked on this project in GSoC 2011. After the summer ended, I spent months seeking a job so I didn’t pay much attention to the project at that time. And later, I made several commits to the project and it’s looking better.

So far, we can generate correct metadata for 13849 ebuilds (5435 more than before). Nearly half of the ebuilds in the main tree can be handled by libbash now. So the goal of this year’s GSoC is to make libbash be able to handle the rest of the ebuilds.

What’s stopping us from handling more ebuilds? There are several eclasses that our grammar file can not parse. That means any ebuild that inherits from these eclasses can not be handled by libbash. On the other hand, the runtime part is not fully implemented, which is due to lack of grammar support or low priority. I have to say it’s hard to tell all the problems we are facing at once. I have been working in this pattern for quite some time:
1. Use our implementation of instruo to generate metadata for the ebuilds in the main tree.
2. Sort the error output to see which errors are the biggest enemies.
3. Solve the problems that are figured out from the error output
Generating ebuild metadata will take about 10 minutes so this process is not fast. I need to solve as many problems as I can at one pass. Typically new errors will occur after I fix the old ones. And the new errors are not introduced by the new commits. They were just hidden by the old ones. That’s why it’s hard to move forward and handle more ebuilds in a short time.

In future, the error output should be improved so it would be easy for us to figure out more problems. After constantly working on the error output for some time, I’m sure we will get fewer errors and handle more ebuilds. Looking forward to seeing a working libbash for all the ebuilds at the end of this summer.


, ,


libbash weekly report #6

Last week I tried to do semantic predicate in our parser grammar. We hoped that it could solve most of the problems we were facing. Unfortunately, it didn’t end up well as we cannot make semantic predicate working with backtracking. This week I will migrate to ANTLR 3.4. and see if it helps solve the problem. Here are what I have done in the last week:

  • Supported braces in command arguments
  • Improved comment handling
  • Supported ANSI C Quoting
  • Supported shortcut capability for && and || in arithmetic expansion
  • Supported arithmetic expression
  • Supported break built-in
  • Improved our build system to reduce dependencies
  • Made arithmetic expansion follow POSIX
  • Improved exception hierarchy
  • Implemented shift built-in
  • Improved the ast_printer utility

A few stories are blocked due to the backtracking problem.

This week I will:

  • Upgrade to ANTLR 3.4 and see if it helps solve the backtracking problem
  • Fix errors in the CI Server
  • Break down walker grammar to reduce compile time
  • Use bash to verify test scripts
  • Support alias
  • Handle options to the local built-in
  • Upgrade to Paludis 0.64.1
  • Support thread-safety
  • Clean up warnings from doxygen

, ,

Leave a comment

libbash weekly report #5

In the last week, I focused on parser grammar improvement. So far we can generate correct metadata for 8028 ebuilds. As we have made error handling POSIX compliant, any parsing failure will cause an exception. So making the parser working properly is the first thing that should be done. To be honest, fixing bugs in the parser is not easy. The logic there is already quite complicated and backtracking make it worse. But I need to get through it. Here are what I have done in the last week:

  • Supported bash redirection for all kinds of commands
  • Supported the special parameter $-
  • Supported parsing -o and -a operators for built-in test
  • Supported brace expansion
  • Implemented eclass parse failure cache
  • Supported backslash escapes inside double quotes
  • Fixed variable indirection in arithmetic expressions
  • Supported regex match operator for keyword test
  • Tried to parse here document and improve variable expansion
  • Improved CI server configuration

This week I will:

  • Support braces in command arguments
  • Improve comment handling
  • Handle single quoted string in variable reference like $’string’
  • Support shortcut capability for && and || in arithmetic expression
  • Support arithmetic expression
  • Support break built-in
  • Support read-only built-in
  • Improve our build system to reduce dependencies
  • Make arithmetic expansion follow POSIX
  • Improve exception hierarchy
  • Implement shift built-in
  • Try boost::spirit::qi to implement a simple lexer for ANTLR

At the end of this weekly report, I’d like to mention a small tip for bash arithmetic. As you probably have known, you can’t write $(( expression )) as a bash command. But sometimes you just need to evaluate the expression and nothing else. You can certainly call the let built-in but that requires you quote your expression to avoid word splitting. Then some people invent this:

: $(( expression ))

‘:’ is a bash built-in that does nothing. So the argument gets evaluated first and then ‘:’ gets called. I really feel it unnecessary as you can always use bash arithmetic expression:

(( expression ))

This is exactly equivalent to

let "expression"

, ,

Leave a comment

libbash weekly report #4

This week I’ve made an important change to bash function implementation. So far we can generate correct metadata for 7927 ebuilds. Here’s what I have done:

  • supported not equals in arithmetic expansion
  • supported array offset expansion like ${*:1:2} and ${a[@]:1:2}
  • improved the implementation for bash functions
  • supported shopt -p
  • supported declare -p
  • supported printf
  • supported $#
  • fixed case statement with empty body
  • removed unnecessary abstractions in the implementation of arithmetic expansion
  • filed new stories from the output of instruo

In the coming week, I’ll:

  • Support bash redirections for all kinds of commands
  • support $-
  • fix problems with built-in test
  • support brace expansion
  • cache parsing failures
  • support indirect reference in arithmetic expansion
  • support escaped characters in double quoted string
  • fix bugs in arithmetic expansion
  • continue working on the story for the CI server

Till now, we have supported most of the language features that are needed for metadata generation. But there are a lot of small problems that we have to handle. The most difficult part is to improve the parser grammar. Even a small change to the grammar could break a lot of things because the grammar is already complicated. So our progress may be slowed down when I have to deal with parser grammar improvement.

Hope we can get more stuff fixed in the next couple of weeks.

, ,

Leave a comment

libbash weekly report #3

We use Scrum in our project. Petteri calls this iteration “getting hotter” because we already have a lot of features working. In this iteration we want to make them work better. Here’s what I have done in the first week (two weeks for an iteration):

  • Supported the eval built-in
  • Supported pattern matching in keyword test
  • Supported simple IO redirection
  • Supported arithmetic expression in keyword test
  • Made error handling POSIX compliant
  • Improved command list
  • Improved variable expansion and arithmetic expansion
  • Supported more Portage functions (ewarn, debug-print, etc.)
  • Cleaned the output of unit tests
  • Improved the quality of the code (explicitly disabled copying, fixed header ordering, compiled our code with -Wconversion and -Wsign-conversion, etc.)
  • Fixed mistakes in the metadata that our instruo generates
  • Examined the syntax that we can not handle in all eclasses and added corresponding stories
  • Added stories from the output of our instruo
  • Wrote a simple ebuild to install necessary dependencies (layman -a qiaomuf)
  • Created a home page based on Gentoo Infrastructure (new Gentoo project)

Before making error handling POSIX compliant, we could generate correct ebuild metadata for about 7800 ebuilds. After that we can only generate 1689 because we will stop if any error occurs.

In the next week, I’ll:

  • Make the CI server running
  • Support array index in variable expansion
  • Add more stories based on the output of instruo

I can’t tell more because I don’t know what stories I’ll add. We’ll see in the next weekly report.

, ,

Leave a comment

libbash weekly report #2

With the effort we spent last week, now we are able to generate correct metadata for 3008 ebuilds. Here is what I have done in this week:

  • supported appending to variables like foo+=bar
  • partly supported the shopt built-in
  • supported the unset built-in
  • supported the continue built-in
  • supported single quoted string
  • supported command substitution in double quoted string
  • supported more characters in function name (not approved by bash manual but used by some eclasses)
  • made the parser write current filename to output if something goes wrong
  • improved test coverage
  • made parser fail if it doesn’t match to EOF

The last thing makes our instruo fail to generate metadata for most of the ebuilds because some important eclasses can’t get fully parsed. We will tackle this issue in the next iteration.
We planned to have our CI server setup. But as robbat2 is on his vocation so we decided to put off the story to the next iteration.
I also wrote an article talking about LCOV and gcov usage in our project. That helped us improve the test coverage.

In the coming week, we plan to:

  • fully parse the eclasses that are required for metadata generation (if it’s not doable in one day, add new stories).
  • add stories from the output of instruo
  • fix our instruo to make the order of INHERITED right (part of ebuild metadata)
  • support pattern matching in keyword test
  • compile with -Wconversion and -Wsign-conversion in developer mode
  • support more Portage functions
  • write ProjectXML for our project

Thank you for your attention.


Leave a comment