How Breakthroughs Happen: Forward and Preface

This book by Andrew Hargadon talks about why some companies are more innovative than others. The strategy focus should not just be “where we are going” but “how to get there”. General insights:

• Creative innovations often come from combining other ideas
• Creativity often occurs within the context of a community of people searching for solutions to a problem
• Tension/trade-off between exploitation and exploration – but this tension does not have to be. Companies can innovate during the process of exploiting existing knowledge and resources.

Chapter 1

• We need to recognize the “networks” that lead to innovation. Not traditional networks, but networks linking people, objects and ideas, all connected to or influencing in some way the final innovation.
• Technology is the arrangement of people, objects and ideas to accomplish a particular goal. In other words an innovation is not just an object — it requires the (possibly specialized) people that use this object.
• Breakthroughs cause these networks experience a shift. Objects, ideas and people can become obsolete. Good inventors are just well connected.

Input Validation or Static Analysis

From a discussion on the SAMATE mailing list: does static analysis add much value for web security or can many problems be found with good input validation. Checking for correct input validation is very program dependent and hence is hard for static analysis checkers. But there are some problems that surface even with good input validation and this is where static analysis can be helpful. Ultimately, the best solution is some combination of “dynamic fuzzing” (feed randomly generated faulty inputs during testing) for input validation and static analysis for everything else.

Breaking up static analysis and defect detection

In preparing a recent survey for FindBugs, I thought about whether to ask questions about “static analysis” tools, or “automatic defect detection”. The two phrases are often used interchangeably, but really both are needed to do the sort of things that FindBugs does. This is illustrated well by GrammaTech’s products:

• Code Surfer is the static analysis part; it can be used for other applications, not just bug detection
• CodeSonar is the bug detection part; it uses Code Surfer and infers bugs from the results of this analysis.

This separation is useful because it means that Code Surfer can be used in other ways, others wanting to do bug detection don’t have to reinvent the analysis wheel, and its analysis is not too narrow (specific to code sonar).

ASE Review

I thought ASE was productive, and it seems this community would be interested in the research we do on static analysis tools.

I talked to Tim Menzies who presented a paper on “The Business Case for Automated Software Engineering” (http://menzies.us/pdf/07casease.pdf). The paper uses AI methods to model predictors for software effort and defect levels in different projects, and shows that tools are helpful even though the models depended on “local tuning” that were different from project to project.

I made it to the talk on “Modeling Bug Report Quality”. The paper is similar to the Nebraska/Google research on the Google bug database, except they evaluate the FireFox database. It might be interesting to compare the two papers.

I also talked to Rustan Leino from Microsoft Research. He gave a good demo of Spec# (a superset of C#) in his keynote talk (http://research.microsoft.com/specsharp/). I think they’ve done a good job of taking (sometimes esoteric) concepts in software verification and putting them in a format that will be accessible to the average programmer, and he pushed the idea of using Spec# for students learning how to program.

Static Analysis and Spell Checking

One perspective is to see Static Analysis tools as “Spell Checkers” (SPSA, 21). Even good writers and spellers need spell checkers. But I’ve never gone out to buy/install a spell checker. I just expect it with my Word Processor, and I lazily expect a red squiggly line to point out my errors even before I run it. So perhaps developers shouldn’t be worrying about static analysis tools but should expect them to be there and to raise flags as they type.

Unfortunately Static Analysis warnings are not as black and white as spelling errors (most of the time).

Findbugs has a feature in which warnings are raised as they occur. We should survey to see how this feature is accepted/used by developers.

Static Analysis User Study

The goal of SAUS research is two-fold:

• Look for ways to improve static analysis tools so that they better support the users’ work and fit into software development processes, and
• Validate/Invalidate assumptions made by tool creators based on the experience of real users.

To this end, we are conducting studies to learn through observation, interview, and survey how static analysis tools are used in practice to improve software quality and security. Some of our activities include:

• Online surveys to solicit feedback from software developers and other users
• On site contextual interviews in which we observe users as they use the tools
• On site retrospective interviews in which we explore in more detail the user’s experiences using the tools
• Literature review to gather the assumptions of experts and tool creators, and the findings of other research projects

Some questions we use to focus our research include:

• What (interface) features help make static analysis tools easier to understand, and easier to fit into a software development process (and the converse of this)?
• What do users like/dislike the most about static analysis tools?
• What road blocks prevent users from integrating tools, or following their recommendations?

The big ideas in static analysis research can be described using a list of problems and solutions. Essentially the purpose of this research is to validate existing solutions and propose new solutions (and problems).

H-1B Visa Proposal

As an international student, I’m always interested in the highly charged debate about allocation of H-1B Visas to foreign technology workers. For one thing, this issue impacts me directly — my first job offer out of college from a major consulting firm was rescinded when the recruiter realized that she had goofed and could not recruit an international student for the position. For another thing, this is one of those great debates that is often presented as a moral issue but, really, everyone chooses the side that benefits them the most. I, of course, support more and more visas, but only because this benefits me — I claim no high moral ground.

And then I learn that many democrats in congress (who I generally prefer to their republican counterparts) are opposed to visa increases and want a tougher system. So how do I solve this moral dilemma: switch parties, or support a side that does not benefit me?

Well I have what I think is a win-win proposal. I propose that congress pass a law that requires all employers wanting to hire H1-B workers to first declare the fair market value for these workers, and then pay them 30% more than the declared fair market salary! This is a win for American workers and liberals because it will weed out any employers looking for cheap labor. And it is a win for me because that 30% could be worth as much as $30,000 in extra income! (Now that I think about it, the same rule should apply for international graduate assistants. I’ll send a note to my graduate student council president.)

I suppose its not a win for the employers. But after all, the official argument is that they have to hire foreign workers since there are not enough american workers. If a company has to do something, then there should be a premium on it, and I should get that premium. If a company has to hire me because I am the most brilliant person they’ve ever met, then there should be a premium on it, and I should get that premium.

And besides, companies can afford it, and its the right thing to do. Whenever my Dad was moved by his company from his home job (in Nigeria) to a foreign job (in Europe somewhere), he always got extra financial allowances to account for the inconveniences of living in a foreign land. Foreign workers have to pay more to go home for Christmas and Thanksgiving; or to mail an absentee ballot to their home countries; some even have to pay taxes in their home countries (e.g. property taxes if you own a home abroad).

I think this is a great idea. I would write to my senator… except that as a non-voting temporary resident alien, I don’t have one.

Running JUnit 4 in a JUnit 3 environment

I recently had the pleasure (sarcasm implied) of upgrading the test environment on the Submit Server. We use the Submit Server at UMD to manage, compile and test student programming submissions. We wanted to support JUnit 4 style test cases, and I decided that rather than rewrite the entire framework we built to support this, I would run the tests in a JUnit 3 environment using the JUnit4TestAdapter.

The first complication was understanding how the environment deals with mixed tests. It turns out that if the test class extends junit.framework.TestCase, it is treated as a JUnit 3 test (@Test and other annotations are ignored). This is also the case in the provided JUnitCore runner, and in the Eclipse IDE. I don’t think this is too well documented (and an early article at IBM suggests this was not the intention), but I decided to keep this behavior and inform our users.

Another issue is that our process requires us to run one test at a time. I eventually figured out the JUnit 4 uses filters to decide what tests to run, and I created a custom filter that does this. The funny thing is that the “filter” method to call this filter on a JUnit4Adapter was not visible (it was private in the code). It took a little while to figure out that this is a bug in JUnit 4.3 and I needed to download JUnit 4.4 (released just a month ago).

Ah the joys of manipulating JUnit. Until my next experience…

Current Research

My current research is based around a few studies of the way Static Analysis tools are used by software development teams. (Here, static analysis tools includes tools for finding programming bugs or security holes.)

We plan to conduct these studies using contextual inquiry — a method in which the researcher (me) visits the site where developers work, and watches them as they program or run tests. This approach should yield details that even developers do not notice as they work.

We hope to identify features that make these tools easier to understand, and easier to fit into a software development process.

Right now, we are trying to identify local companies that actually use static analysis tools. Its proving to be hard to find regular users, but security consultants, and software developers serving high risk markets may be good targets.