Archive for May, 2011

libbash weekly report #2

With the effort we spent last week, now we are able to generate correct metadata for 3008 ebuilds. Here is what I have done in this week:

  • supported appending to variables like foo+=bar
  • partly supported the shopt built-in
  • supported the unset built-in
  • supported the continue built-in
  • supported single quoted string
  • supported command substitution in double quoted string
  • supported more characters in function name (not approved by bash manual but used by some eclasses)
  • made the parser write current filename to output if something goes wrong
  • improved test coverage
  • made parser fail if it doesn’t match to EOF

The last thing makes our instruo fail to generate metadata for most of the ebuilds because some important eclasses can’t get fully parsed. We will tackle this issue in the next iteration.
We planned to have our CI server setup. But as robbat2 is on his vocation so we decided to put off the story to the next iteration.
I also wrote an article talking about LCOV and gcov usage in our project. That helped us improve the test coverage.

In the coming week, we plan to:

  • fully parse the eclasses that are required for metadata generation (if it’s not doable in one day, add new stories).
  • add stories from the output of instruo
  • fix our instruo to make the order of INHERITED right (part of ebuild metadata)
  • support pattern matching in keyword test
  • compile with -Wconversion and -Wsign-conversion in developer mode
  • support more Portage functions
  • write ProjectXML for our project

Thank you for your attention.


Leave a comment

Use gcov and lcov to know your test coverage

As we all know, unit test has been a crucial part of software development. I feel I was walking on ice if I was coding without writing tests. But even if we have unit tests, things may still go wrong if the tests have bad coverage. Fortunately, there are many tools today to help us analyze our code and give us every detail of the quality of our unit tests. GNU gcov is such a tool for code compiled with GCC.

gcov is very easy to use. If you’re developing a C/C++ project, all you have to do is to compile your project with “-g -O0 –coverage”. “–coverage” is a synonym for-fprofile-arcs, -ftest-coverage(compiling) and-lgcov(linking). This tells the compiler to generate additional information and code needed by gcov.

OK, let’s run our tests. GCC will generate a bunch of files(profiles) that will be used later. These files know how your tests was executed and record many information such as arc transition counts and some summary information. Now we’re ready to view our test coverage now with gcov.  But gcov doesn’t give us a report that is easy to read and it’s not convenient. Here comes the tool called “LCOV”.

LCOV test coverage report for libbash

Read the rest of this entry »

, , ,


libbash runtime weekly report #1

The following content is copied from my weekly report email in gentoo-soc mail list

Hello everyone,

This year I will work on the libbash runtime project. A short introduction for this project:
Libbash will enable programs to use Abstract Syntax Trees(AST) to parse and interpret *shell* scripts directly instead of using regular expressions. Most of bash 3.2 syntax will be supported. This will be a great benefit to programs both outside and inside Gentoo, including Portage/Paludis and repoman.
For more information, I wrote an introduction[1] explaining its potential usage and benchmark. There’s also a home page[2] for this project. You can find out the rationale, plan and detailed progress there(most content comes from my GSoC proposal).

As the first weekly report, I’d like to briefly talk about our current progress. I’ve been contributing to this project since March , so we have done a lot of things. Here’s a summary:

  • Parser grammar improvement
  • Build system improvement(C++ hidden visibility, developer flags, gcov, etc.)
  • Shell arithmetic
  • Variable(string, array) definition and reference
  • Variable expansion
  • Shell pattern matching
  • Compound statement(if, for, while, until, case)
  • Shell test expression
  • Command execution and substitution
  • Shell function
  • Several shell built-ins(source, let, etc.)
  • Utilities(ast_printer, reimplemented version of Paludis instruo, etc.)

For more detailed information, please visit our home page[2], my blog[5] or our git repository[3].
Here are some resources we have:

  • CI server(not public accessible yet)
  • Agilefant server(for Scrum)
  • Test coverage report
  • Callgrind and massif target
  • Github repository[4] (latest commits)
  • Canonical repository[3] (reviewed commits)
  • Home page[2]
  • #gentoo-libbash IRC channel

Now we can generate correct metadata for 2934 ebuilds(There are 27289 in total). We will get more and more during the summer.


, ,


Home page and updated benchmark for libbash

I’ve created a simple home page for libbash. From there you can see the general design of libbash, Q&A and the detailed plan. Most of the content comes from my GSoC application. I’ll blog our latest progress here and update the home page during the summer.

The benchmark mentioned in my previous post(Introduction to libbash) is updated to be more convincing. Thanks to nirbheek and Ford_Prefect for telling me about the kernel cache. Thanks to ferringb for telling me to add a test for pmaint (The fastest metadata generation tool for now based on the test result). I also added more information about the testing environment including the CXXFLAGS I used to compile Paludis and libbash. This time I removed -D_GLIBCXX_DEBUG (libstdc++ debug mode) and the speed of our implementation is doubled.

For people who are interested in this project, we’ll have a demo on IRC channel #gentoo-libbash at 2011-05-16 06:00 UTC. Please feel free to join.


Introduction to libbash (GSoC 2011)

In this article, I’d like to introduce the libbash project. I’ll explain what libbash can do with some examples. At the end of this article, a benchmark result is given for egencache (Portage), instruo (Paludis) and instruo reimplemented with our library.

I’ve been planning to write this article for a long time. This project was started last year as a GSoC project proposed by Petteri Räty. Nathan Eloe did a great work and achieved to build Abstractr Syntax Tree(AST) for a give shell script. This year, again as a GSoC student, I’ll work on the runtime part, or put it simply, making the library capable to run shell scripts. I’ve started contributing to this project since March 2011 and find it really amazing.

Libbash will enable programs to use Abstract Syntax Trees(AST) to parse and interpret shell scripts directly instead of using regular expressions. Most of bash 3.2 syntax will be supported. This will be a great benefit to programs both outside and inside Gentoo, including Portage/Paludis and repoman.

For instance, you have /etc/conf.d/net which is essentially a shell script. Libbash will tell you what variables and functions are there, what values of the variables will be after interpreting it. It also allows you to use compound statements and shell builtins in the script. We plan to support common bash 3.2 syntax except features related to interactive shell and executing external process. Currently it lacks a lot of functionality ( of cause 🙂 ), but it begins to shape up and can do some real work.

Let me show you how we handle /etc/conf.d/net with the library at hand.

$ ./variable_printer /etc/conf.d/net

The variable_printer is a utility program that is linked against our library. All the non-local variables defined in /etc/conf.d/net are printed out by the utility including arrays. Actually we can do much more than that. For example, function definition, variable expansion and command substitution are supported(Although their functionality is not complete yet). If you need to analyze bash script, this library should be helpful.

Read the rest of this entry »