Week 8

This week was a slightly less productive week.  I made one or two tweaks to the image diff tests, but otherwise, they’re done.  They can be downloaded here: http://sourceforge.net/projects/blenderrender/files/

Although Andrea only looked at my code to determine good structure, etc.   I believe that most of the bugs have been removed.  (Although, this is python, and bad code doesn’t show up until someone tries to run it, and I haven’t made regression tests for my regression tests. (at least not yet anyway, and it’s not on my radar in the near future)).

Overall, I think this has been a good week.  I worked almost exclusively on the image comparison diff tester this week.  I wrote it in python, and it can actually work independent of blender (although it does assume that blender is on your system, and in your path, although that can be changed with the –blender-bin flag).  This means that the tests folder can be put in a zip, and placed in the same location as where the current regression tests (for blender 2.4x) are located on the website.
As I said last time, the general way to run the tests is by using python run.py.  You can use the -v flag to see the output of blender as it’s rendering it.  Also, it might be possible to avoid even having to download the images at all, by using the recently built –built-tests flag, which, instead of rendering tests, will render the images and store them in the known good folder.  (It uses the version of blender in your path).
Through the use of the –image flag, you can compare any blender image on your harddrive (currently assuming it’s in the right file structure), this allowed me to make other image comparison tests, that compared blend files built by pyunit based tests.  This (along with the hashcode operator I started putting together last week), means that most of the tools for building good unit tests (and regression tests in general), are almost build (I think).  Meaning that now the only missing piece of the puzzle is the one I was working on several weeks ago (without luck), which is:A.  Getting the python API to work in background mode (which blender currently isn’t designed to do).B.  Getting the python API to work properly in foreground mode (lots of stuff in bpy.wm crash.  (At least it did as of a week ago, do to low bandwidth caps here, I won’t’ merge until tomorrow evening (when I get home), so they may have been fixed by now)).

Although Andrea only looked at my code to determine good structure, etc.   I believe that most of the bugs have been removed.  (Although, this is python, and bad code doesn’t show up until someone tries to run it, and I haven’t made regression tests for my regression tests. (at least not yet anyway, and it’s not on my radar in the near future)).
The comparator is also capable of producing animation tests.  However, this code is slightly buggier.  Also, by default, I’m not running the animation tests in the current regression suite, because it takes the running time up from 2:30, to over an hour.  I have a –animation flag, but it doesn’t work at the moment.  (Although the underlying code works, so it’s basically just a matter of plugging the code into the flag).

Finally, because the animation images make the overall file size ~240 MB (as apposed to 30 MB), they are not in the default build.  But rather, the user has two choices, they can either A:  Render Good ones, or B.  Use a hash code (and download images (or generate them) if something goes wrong).  The hash code also work in images.  However, the code for this is only half built.  The test case classes support it, but the surrounding structures don’t use it yet, this shouldn’t be more than an hours worth of work to implement it fully.

Finally, I didn’t do as much work as I would have liked in the HTML output.  Now, rather than using static CSS scripts from sphynx, I now build the CSS in the script itself.  (With the exception of images, I’m not sure how to do that as a python script…well, there is PIL, but I’d sooner put bamboo up my fingernails than make the needed images that way).  Although the output is starting to look like it’s from blender’s website.  (Although the table doesn’t quite fit).  Also, it will only generate images for single images (well, that, and animations, but it only does the first frame).  Eventually, the animations will be clickable, and you can view a separate HTML page for the entire animation, but that’s not built yet.  It shouldn’t be too hard to make, but I keep having to do one or the other thing, so the current implementation is old, and would probably cause the script to crash if used.

Render Tests

The current set of render tests requires PIL, and python 2.x. In order to run them, inside the blender source directory, look in: tests/render. There, you should find a collection of files. While the absolute path of those files doesn’t matter (aka, you could move the tests/render folder anywhere, and it would do just fine), those files must be kept in the same relationship to each other. If blender is already in your system path, just go into the folder, and run python run.py, and the script will take it’s course. If blender is not in your current path, open up run.py, and change the BLENDER_BIN variable to you blender binary, and than run python run.py. You can view the results in the console, otherwise you can see a list of all of the results, as well as diffs, by opening index.html in your web browser.

(my most recent commits have temporarily broken this as an operator, but I will soon fix that).  There is a webpage that is outputted which allows you to easily see the difference between the renders.  In addition, the algorithm for analyzing the images is much improved.

I also moved my tests.hashcode module into an operator (bpy.ops.tests.hashcode()), or Tests->Hashcode.  Although it still only gives feedback in the terminal, and only takes some things into account when making the hashcode.

I’ll try to make this short, as it’s late here.
Even though I got sick this week (it turned out to be just a cold/flu), and went out of town (and still am), I believe that this week has been one of the most productive weeks so far.
This week started off with me implementing some of the GUI features I worked with last week, which is to say I started making the image diff tests into an operator.  (Previously they had been baked into the tests directly).  They could be run from a Tests dropdown menu, although there was non feedback outside of the console, leading the user to believe blender had frozen while the tests were running.  Andrea pointed out that this was pointless, and that it should just be a script.  As such, I started implementing it in all in python 2.x.  The tests can be found in tests/render, and I put together a slight howto on how to run them: http://wiki.blender.org/index.php/User:LeifAndersen/GSoC2010/Howto
Render TestsThe current set of render tests requires PIL, and python 2.x. In order to run them, inside the blender source directory, look in: tests/render. There, you should find a collection of files. While the absolute path of those files doesn’t matter (aka, you could move the tests/render folder anywhere, and it would do just fine), those files must be kept in the same relationship to each other. If blender is already in your system path, just go into the folder, and run python run.py, and the script will take it’s course. If blender is not in your current path, open up run.py, and change the BLENDER_BIN variable to you blender binary, and than run python run.py. You can view the results in the console, otherwise you can see a list of all of the results, as well as diffs, by opening index.html in your web browser.
(my most recent commits have temporarily broken this as an operator, but I will soon fix that).  There is a webpage that is outputted which allows you to easily see the difference between the renders.  In addition, the algorithm for analyzing the images is much improved.
I also moved my tests.hashcode module into an operator (bpy.ops.tests.hashcode()), or Tests->Hashcode.  Although it still only gives feedback in the terminal, and only takes some things into account when making the hashcode.
As far as next week goes, I am going to spend a bit of time with the webpage.  It’s not quite flush with blender’s webpage, but it’s getting closer.  But more importantly, I can see a few more options that would allow the user to get more of the raw data behind the images (which I would assume would make it debug).  Most of this I believe can be written in HTML, but some javascript may be useful.  I also plan to get animations working with these tests, and possibly bringing this back in to work with the blender operator.  I also am going to try to get a lot more feedback this week, and hope to have the tool used soon after.  I will also spend a bit of time getting the hashcode operator to take more into account, as well as giving more feedback to the user.  (As in feedback that’s not only just console output).  I would also work on aggregating the data in the blend file for the user to test, but blender is already capable of that.  Finally, I plan on improving the integration of Ctest and CDash, but that seems trivial to do.

This week has been one of my better weeks.  As usual, there were a few bumps this week, but not too many.
This week dealt with two things:1.  Image comparison2.  UI, and thus wrapping unit tests in operators to insert into the UI.
Per Matt Estela’s recommendation, I tried incorporating Perceptual Image Diff http://pdiff.sourceforge.net/ into the tests to compare images.  The pre-compiled binaries work fine, however, compiling from source presents a bit of a problem, even with all of the listed dependencies installed, it complains that several printf stamens are not declared.  I emailed the developer and am waiting for a response.  This isn’t my main priority.  However, I will keep using PIL until such time as a tool like this works.
Getting tests in blender’s UI represents some challenges that do not exist when using CTest as a testing platform.  First of all, I thought a lot about render tests.  If we use blender’s current set of regression tests that blender has, it shouldn’t be very difficult to create a tool that allows a developer to render a blend file, and than compare it to another image file, and than write another tool to run all of the tests.  The problem is that putting this into the GUI, which requires python 3.  And on top of that, developing a good UI for this sort of thing can also be a bit tricky.  Fortunately though, it still is trivial to wrap up unit tests inside an operator, so ultimately, running the tests won’t be too big of a deal, as apposed to setting them up.  I also toyed with writing the operators required in C, although this isn’t as good of an idea as writing it in python, where we could simply give developers a template file to use for their tests, but it may still prove useful.
Next week should be interesting.  Starting on Wed.  and going to the end of the week after that, I’ll be out of town.  With that being said, I won’t be out of communication, but rather, I’ll probably be spending less time on IRC.  (I’ll still try to attend the usual Sunday meeting, as well as spending an hour or so online though).  I will respond to emails as usual, albeit probably a bit slower.   I’ll post an email specifically about this to this list in case this portion of the review is missed.  I still have more UI stuff planned next week, but mainly merging the UI and rendering algorithms into one fluid tool, at which point I’ll start automating it.

So, just a brief checklist of what has happened:

1. Project Started
2. Gtest set up
3. Pyunit setup
4. tests module created
5. Hashtests and image comparison tests built
6. Learned a lot about how the Blender codebase works
7. Made some hacks and bug reports with the Blender codebase
8. Initial work on GUI panel for tests (originally dropped, but recently brought back into the foreground)
9. CMake now using CTest and CDash to report the project

There was probably some more, but those were the biggest.  When I look back on that, my initial impression is that I really could have gotten more done.  In particular, most of this stuff is samples, I have been working on the bones behind this framework, and the metaphoric meat has yet to come, although it did start to at the end of the month.  However, it than becomes important to look back at the schedule planned for this project.

At the time of writing this, the predicted portion of work is supposed to be:

June 8 – June 23

This time should be dedicated to development of other end to end and regression tests. It is much longer than the first suite because much less time was spent on it during the community bonding period. An emphasis will be put on making various forms of render based tests. At this time, it will be important to determine a good strategy for how often the test suite should be run, and, if needed to be run in multiple increments, i.e. some tests running with every commit, while others run only nightly. This section can be further split into two sections. The first being setting up the testing platforms, and the second part being creating a sample test suite for the built platforms. Note that the task of scouting for viable platforms, based upon what Blender currently uses, and the toolset others, should be a ‘back burner’ project, while the other sections are taking place, and should be completed by the time that this section starts.

So, as far as the project goes, it looks like I’m on schedule, with one exception, I have very few tests in hand, do to the amount of time I’ve spent trying out testing frameworks.  I don’t think this is a very bad thing per say, not only was that particular timeline fairly preliminary, but it also gave a lot of time for testing frameworks.  So, in order for me to stay up to date, this next month will be spending a lot more time developing tests, and a lot less mucking about with how they are presented, especially as most of the frameworks have already been set up, with two exceptions, registering it as GUI within blender, and allowing for easy automation of new tests (currently a developer has to manually plug their tests into the system, at which point they will be automated).  Also, it might be a good idea to make the pyunit tests operators in the near future, which should allow for them to be easily made into a gui.  With the addition of the current regression tests that Blender has, this could be vary useful.

As far as participating in the community, I believe that that is going well.  I’m idling in the channel, and have had several conversations about problems with the code.  At least, I think I’ve done a good job, this is one of the situations where unless someone is telling you that you need to do better, it’s hard to know that you can.  I’ve also participated via email and the bug tracker.  I don’t see a need to make many changes for next month.  With that being said, for about a week and a half, possibly two and a half weeks next month, I’ll be out of town.  I’ll still be working on my project, and responding to email, but do to poor internet access, I’ll likely not be doing too much idling on #blendercoders.  Although I’ll try to be on for an hour or two a day in the evening (CEST time, morning for me in MDT).

So, in short, I believe I’m still in the window that makes me on track.  I’m not at the front of the window per say, but it’s not anything I think I need to worry about, yet.  Also, I’ve been speeding up, so hopefully I’ll get ahead of my proposed schedule soon.  I will be spending more time developing tests, and than plugging in blender’s current regression tests (probably by using the same file structure those tests are currently in), and I’ll be keeping my current pace in the community.

I spent a lot of time this week trying to get some of the tests to work in background mode. Unfortunately, several of the needed api calls (saving/loading/rendering/saving images), only work when in regular mode. I also had other problems with API calls not working properly even in the foreground mode, and I’ve done some more hacks on the codebase, and submitted some bug reports, although at this point, I’ve got most of the API calls I need working, albeit they’re still a bit buggy. In the end, we’ve decided to run the tests in the forground (for now), and just run the exit blender api call at the end of the test. The largest problem with this is that if the test segfaults, blender won’t quit, and the user must quit it himself. Also, it’s quite annoying to have a window pop up and close over and over again while running the tests. Thus, as soon as some of the bugs I’ve reported are fixed, I’ll be switching back to background mode.

I’ve also started doing some image comparison tests. I don’t really have any good algorithms yet, so I’m still very open to suggestions. Currently though, I’ve tried several iterations of a histogram comparison. Both comparing all the color bands at the same time, and comparing them separately, the first one is quite a bit faster, although I think that’s do to how I’m using the API calls.

By far, the largest problem I’ve run into image testing (besides there being no python API call to render an image and save it, even in foreground mode), is the fact that PIL currently only supports python 2.x. So, to compensate for this, I’ve split the image tests into two parts, the rendering test (and everything else that can be done in blender), and the comparing tests, which use PIL. If I need to, I suppose I could read in the files myself, and use the raw data the files bring in, but I believe that would waste time, and wouldn’t be as effective as using something like PIL. The issue than becomes the lack of not being able to render anything and save it in python. To get around this, I’m currently saving to a .blend file, using the command line to render the files, although I’ve run into the problem of getting CTest to run multiple commands in one test, which I’m looking into. Another thing I’m thinking about doing, is getting an exec function for the image.save_as() operator to work. Oddly enough, it only works when the image screen is up (which, in my opinion, goes against blender’s non-modal philosophy), or rather, when the image is in the buffer, (which only happens after the user hits f11 or f12 (+ ctrl, or whatever you set to render/show images/animations), and the problem comes because there is no python API call for it. Oddly enough, in regular mode, the invoke function also works just fine as an exec function, but in background mode, it segfaults. I’ve stopped it from segfaulting, but because the value is still null (do to being in the background), it’s still imposable to actually save the image. (Although the ‘view’ is the image view). Also, the invoke function still is a bit quirky as an exec function. So, I’m thinking it might be possible (because I’m currently running the tests in the foreground anyway), to just render the images and save them in the python file, which would not only be easier (api call wise anyway), than mucking about with CMake, but make more logical sense, and allow for easy unification of the tests when PIL is supported for python 3.x.

Most of this I’ve done to test what appears to be rendering problems with viewing hair as ‘object’ or ‘group’. (Which may have been solved, or proven to be a feature rather than a bug, I haven’t checked the bug tracker for that particular bug for several days). But, as long as it’s not a feature, it still would be useful for a regression test.

I did get a chance to look at sphinx documentation, although not very much, and not enough to get anything done. I doubt I’ll get to it next week, but I still am keeping this as a back burner project. (Albeit a very back burner project, it’s currently more important to me to get the python api up to snuff in background mode).

Finally, today and/or tomorrow, I plan on writing a month in review. Because I like wordpress’s editor, I’ll probably first put it up on my blog (http://leifandersen.net), and shortly post it on my project’s wiki page: (http://wiki.blender.org/index.php/User:LeifAndersen/GSoC2010)

Next week I do not plan on dealing to much with issue of background mode, except as a back burner project. I have submitted bug reports, and will be participating with them though. I may devote a few hours next week to this though. I also plan on finishing up good image comparison algorithms, and I’ll put them in my tests module, however, I won’t be able to use that module until PIL works on python 3.1, but it should make it easier to move the tests. Except for possible rendering problems, I still believe that hashcode based tests are more effective than render tests, so I’ll probably work on that a bit. Finally, I plan on making the structure cleaner. I may remove the gtest portion altogether, as it’s not needed, and just in the way, as well as possibly changing the structure of the python tests to a more logical order, rather than simply module level order. As well as finding a good place to put the CTest commands.

This week I started working with tests for operators as apposed to directly with data. The first problem I ran into was the immutability of bpy.context. I was resorting to hacks to change the context, which only was working some of the time. Thus, I went to GTest, giving it more time than I had planned on this week (a little over a day), because the C-API calls took in a bContext, rather than having it be implicit. However, I never managed to get the linking right, any probes I put into the kernel or operators itself would result with many errors, even for functions that seemed to only allocate space, thus I never got a bContext made.

Fortunately ideasman42 pointed out to me that I could pass in a dictionary, and the operators would use that instead of the value in bpy.context. I haven’t gotten it to work in all of the situations I’ve tried (yet), but many have worked, and it does seem promising. However, that only solves some of the tests, it still would be valuable to run tests using bpy.context, or some other type of implicit context, for better code coverage.

One thing that I really didn’t expect this week, was that most of the bugs I was trying to debug this week, were ones that I found. I was planning on looking for bugs on projects.blender.org, and I did find a few, but when I started building python scripts to test them, I found many more simple problems that I had to deal with first, which took most of my time.

My plans for next week:

My first order of business is to get several operators in bpy.ops.wm (mostly found in the windowmanager folder), working in the python console, and in a script (both when blender has a visible, and no visible window). Currently, I’m working on bpy.ops.wm.read_homefile(), and I’ve made some progress. However, the problem is that the bContext that is passed into it has several null elements in the console, which is expected to be non-null for windowmanager operations. Once I’ve finished that, I will finish my setup.py test, and proceed to use that as my setup method, at which point most of my tests module will become superfluous, and I may take some time dealing with that. In fact, I could likely get rid of it, with the one exception of me using it’s hashcode function. I may also read in a pre-built empty file, rather than the default .b.blend file, which could possibly break the tests if the default was changed, which would be very bad. Finally, as a side project, I’m going to start reading up on how the python API’s documentation is made (I know it’s using sphinx, but I don’t know how it works, yet), and I’ll see what I can do to start helping minddrones with that.

1. The biggest thing is that I believe I now have enough of a framework set up to begin non-trivial tests, which I began to do today, at least in the sense that I started tracing actual bugs, so far to no success.

2. We decided to put GTest on hold, and perhaps drop it altogether, mainly because it’s effectiveness, given time constraints is minimal, do to how the blender kernel works). It’s still in the codebase though, although it does cause some rather odd errors (as in cmake has to be run twice before a successful make will happen).

3. The starts of a hash changer has been posted, although it’s not too integrated yet. I was trying earlier this week to actually build a GUI for it, which could be integrated into blender itself, but this proved futile (thusfar), do to several lacking api calls in blender’s GUI api calls. Still, I think it would prove valuable to add a control panel for that sort of thing into blender itself.

4. Actual support for cdash, with a demo cdash created, type in make report to send the results of the tests to the public cdash: http://my.cdash.org/index.php?project=Leif-Blender hopefully, we can set up Blender’s own Cdash for this sort of thing at something like cdash.blender.org.

5. Removed the superfluous hashtests folder, and integrated it right into pyunit, the tests still exist in there.

6. Removed the binary directory’s dependence on the source directory for running tests, it will now work just fine without the source directory.

7. Created a howto page, in order to help people set up and run the code: http://wiki.blender.org/index.php/User:LeifAndersen/GSoC2010/Howto

8. Got an account at graphical, and started uploading pre-build binaries, and linking to them from the howto page. Currently though, it’s only for 64-bit linux. I’m planning on uploading 32-bit linux shortly, and hoping Andrea will upload a windows build. If anyone also is willing to make a mac build (partially to see if it works), that would be helpful.

Currently, my plans for next week consists of bug-squashing on projects.blender.org. In particular, I intend to create pyunit based unit tests to aid in bug tracking, and while doing that, begin to make a nice little suite of regression tests, to ensure that those same bugs don’t crawl back. Also, most of the tests are still in bpy.data, in intend to do a lot more with bpy.ops next week. I also hope to spare a little time for gtest, but I won’t give it more than a few hours (less than a day’s worth of time), in order to see if we can salvage it, or at least leave it there as part of a framework for building larger tests in the future.

I’ve been thinking for the past few days, everyone’s been complaining about facebook, and my opinion has been: “it’s your fault for trusting some random company on the internet with all of your private information, don’t put information on the internet you don’t mind the rest of the world seeing”.

And then I got to thinking, what about Google, no one ever talks about that, and the amount of data we’re giving Google for email. I then think that it’s not a big problem, because you’re emailing people anyway, so it shouldn’t really matter, after all, they could always go publish your mail…so it’s just better to not say anything that could be incriminating over email as well. Also, I’m sending unencrypted messages, so it’s still technically wide open.

But then I start realizing that it’s still a bad thing. True, while no one at Google is reading my email (I think), who says that they will always be the ‘good guys’, which is to say, just like how everyone trusted facebook, and their motivations changed, nothing is preventing google from changing it’s opinions. And, even if Google keeps it’s slogan, and never does become ‘evil’, they could still get subpoenaed, etc. etc. In short, I’m saying I shouldn’t trust this thing with the cloud.

So, this lead me to think, what is there out there besides GMail? Sure, there is clients like evolution or thunderbird, but they fail short. In particular, they don’t really have threaded conversations. Most of the other things I need (tags, filters, multiple email addresses), either come built in, or a solution can easily be hacked, but I haven’t found any way to thread emails, and with the way I use email, I really need conversation threading. Also, it would be nice if I could hide quotes by default…but I think there’s a solution to do that.

Does anyone know of a good alternative to gmail that will thread conversations. Preferably an open source solution. I know outlook will do it for you.

Thank You.

Update:

Okay, I just found out that evolution does have message threading, but rather than being in the preferences panel, it’s located in the View drop down menu.  Apparently Thunderbird does the same thing.
I also tried out Zimbra, and other than that it appears to be based very heavily on java, it does look like it would also do the job, in addition, it’s also really easy to configure (unlike evolution which requires you to know how email works to configure it, Zimbra’s desktop client will let you simply put in you’re credentials to any webmail based service, or to your own server, and it will set the rest up for you.  Although, it automatically downloads your mail, which could get annoying.

I really love open source, but sometimes, it falls flat on it’s face…this is one of those times. I got a sony vaio vpc eb11ffm from best buy a few weeks back. I brought a livecd, and put it into any of the machines I was considering buying, to see how well it would run. It worked well in the vaio, except for one major flaw, the sound wasn’t working. (I found another one later after I bought it, the system wouldn’t be stable if I had compiz on it).

Anyway, the thing even comes with a version of linux on the machine, for ‘quick boot web surfing’, and the sound works in it just fine. Also, I even was able to find a fix for the sound. It’s not do to any lack of drivers, but one, just one configuration error that ubuntu makes when it’s installing. I go to report the bug, and I even provided the fix to go with it, and it was rapidly merged with another similar, but slightly different bug, and was pronounced to be fixed. I upgraded to the RC that was created today, and nope the problem is still there. And short of opening another bug report, with the same problem, I can’t think of anything to do…grr…

Yes, I know canonical says not to run 10.04 on a machine you need to be stable, but I don’t have much choice. For some reason, 8.10 won’t even boot onto the machine, and I don’t want to go back to windows.

I know you people try hard, and I thank you for it. But sometimes, I just have to rant in frustration, I want my sound to work out of the box.

Thank you for your time. And I’m sorry if this is in the wrong spot.

If you want to see the original web page, it is located here: http://ubuntuforums.org/showthread.php?p=9161218#post9161218