Driven by the success of genomics and proteomics, computational tools are becoming an ever larger part of many corners of biology. In specialties with a strong genomic interest like Evolutionary Biology, understanding of computational tools for database searching, sequence alignment and phylogeny is starting to become requisite knowledge for biologists. However, many biologists see computational tools as peripheral to their work--a means to an end. Accordingly, biologists are motivated to learn "just enough" about a computational tool to perform the operations necessary to get the results they need. This results in improper tool use, which can lead to erronous results.
There is another trend in computational biology towards large, unified databases like NCBI's !GenBank (Benson et al, 2005) or the Gene Ontology (Ashburner et al, 2000). Many of these databases accept contributions from a wide range of sources and are unvetted. In some cases, most or all of the contributions to these databases come from experiments conducted entirely in silico. In the case of the Gene Ontology, 95% of the term assignments were inferred from sequence similarity or other computational analyses (FIXME: personal correspondence). These results are then recycled, as newer research projects use data from these databases to infer additional biological relationships. As a consequence, errors made in computational analyses can be dangerously amplified.
Compounding this issue are the difficulties the field of computational biology is facing with peer review. Journal submissions often come with "60-plus pages of supplementary data", while accompanying data sets "may contain millions of discrete data". (Nicholson, 2006) This volume of experimental data make it difficult for reviewers to assess the validity of computational results.
In addition, biologists seem to be highly constrained by the limitations of their knowledge about computational tools. It can take years for biologists to identify an optimal toolchain and to learn to use it to its fullest extent, if they reach that point at all. Typically, biologists simply make do with what they have: a handful of tools and a subset of their capabilities that were introduced to them through their social network.
There is clearly a strong need for improvement in biologists' understanding of how their software tools work and what tools are available to them.
We conducted interviews with biologists in six labs each with a different biological focus. While there were differences in what software was used and how, there was much in common between the relationship between biologists and their software. Most biologists seem to treat their software as a black box. Their understanding of the function of the software is almost always couched in terms of what goes into it or what it produces as an end product. When pressed, few seem to know any of the details of those ends are produced.
We observed one particularly pronounced example while observing a research associate create phylogenies using Mega. As he started to demonstrate how he conducts the bootstrapping process, he admitted that he did not know what bootstrapping was, or what the difference was between different methods. This sort of situation is not an isolted occurance. Another researcher who uses ARB did not know the difference between the different models it uses, although she knew that it was important that the right model was chosen for each sequence. She saw this as a selling-point of ARB: that it would choose the model for her.
Biologists responses to this situation can be somewhat unsettling. In many cases, it seems that biologists rely on self-perpetuating institutional knowledge: they simply ask their lab partners how they use the tools and then imitate their workflow. In other cases, a biologist might try multiple parameters then compare the results. If the results appear identical, the biologist will assume that the parameters do not matter, and it is safe to choose either. This is a dangerous assumption: different data might not be as forgiving. Worst of all, in some cases biologists will try multiple methods for achieving some output, compare the results, and then choose the result that looks most correct to them.
These practices clearly raise ethical questions about scientists' responsibility in reporting computational results, but it also represents an opportunity for exploring ways to help biologists to become more familiar with how their software works.
The most obvious way to improve biologists' understanding of software is to provide better documentation. Many popular web-based bioinformatics applications now include inline help, which allows biologists to click on terms used in the interface and read about them. Because this documentation sometimes assumes a high level of understanding, or is otherwise unhelpful, biologists who want to answer a specific question they have about the software often have to engage in a broader search for information on the web. This is a presents aprohibitively high barrier to entry in learning.
A similar problem exists with the many educational resources available to biologists. There are dozens of interaction demonstration tools for sequence alignment alone available on the web [1,2,3,4,5]. We presented one of these, !BiBiServ's Sequence Alignment Applet, to a masters student and a postdoc in biology, and in both individuals found it largely incomprehensible. In addition, finding these applets is difficult. A biologist would need to know that they are called "applets" and include that terminology in a web search. Again, we found that this is a level of committment that biologists are not willing to invest unless it is strictly necessary for getting results.
Underbelly is a sequence alignment tool we built as a showcase of a different approach to bioinformatics education. Rather than providing external documentation or separate educational materials, Underbelly exposes the innerworkings of the sequence alignment process to biologists during their normal use of the tool. Rather than presenting users with a progress bar and then results, Underbelly shows the results being calculated (at high speed) and provides the ability to pause the process, step forwards and backwards through the algorithm, get and more details about how different parts of the algorithm work.
There are five critical aspects of our design:
First, Underbelly makes exposure to the innerworkings of the algorithm unavoidable. The biologist is free to ignore what is happening onscreen between the time they enter their data and the time they see the results, but they must at least register the fact that some calculation is happening and that it is at least somewhat accessible to them. The interface is presented in as friendly a way as possible, so that biologists can see how easy it is to step inside the algorithm and see what is happening.
Second, the algorithm is presented at multiple levels of granularity. By default, the algorithm is represented in high-level pseudocode (Figure 1) which is readily understandable by biologists. No initiative is required to read this overview while the user is waiting for the algorithm to do its work. However, if the user's curiosity is piqued, they can click the "Show me" button next one of the stages, and a more detailed representation of that particular subroutine will be shown (Figure 2). In this way, Underbelly offers both a very low barrier to entry, and the opportunity for a very deep kind of educational exploration not currently afforded by most documenatation.
The third key element of our design is that the user can explore the execution space of the algorithm very freely. Not only can they play and pause the algorithm, they can step forward and backward through it, jump to specific points in their data, and move between different stages of the algorithm. Eventually, we would like to provide the opportunity for users to modify the data being manipulated and watch results change.
It is also important to note that Underbelly works with a user's own, real data. Although the current implementation is a research prototype, and thus somewhat simplified, it is not meant to be a toy application used only for educational purposes. It is indended to be a real, usable application that provides integrated learning and exploration. It is our hope that such a configuration will overcome the barrier to entry problems of current documentation and educational materials.
Lastly, Underbelly does not provide any representations that are unique to the sequence alignment process. Many of the existing interactive sequence alignment demonstration programs represent the sequence alignment process with a special visual language, representing the grid as a field of arrows, for example[1]. We wanted to design Underbelly in such a way that it might feasibly be generated automatically from code that had been annotated with pseudocode descriptions and some hints about how to break it into meaningful steps. We are proposing Underbelly as a template for a new way of presenting software to biologists, and as such we wanted to make it as generic as possible. Generating an Underbelly-like application from source code is a non-trivial task, but we wanted to make sure that it was at least possible in theory. Future work will explore what would be necessary to make this happen.
Our goals for the design of Underbelly were to provide immediate educational payoff to biologists who want to invest a small amount of time into learning about the software, and to encourage and support critical thinking about the function of software. We began our design process by doing a series of observations and interviews in four biology labs, interviewing researchers about their research goals, their tools and their workflow. Based on this research and discussions with bioinformaticists, we identified Sequence Alignment as an important computational task in biology that is simple enough for us to build a prototype around. We began with simple sketches which were eventually expanded into a paper prototype that could be used to test our design.
The fundamental difficulty we encountered with our initial evaluation attempts was that it is extremely difficult to use a "Wizard of Oz" prototyping method (Wilson and Rosenberg, 1988) that allows users to freely explore an intricate computational process. Each time the user pressed the "next" button, the human who was acting as the "computer" had to move several bits of paper around, draw a number of figures on transparencies, look up data in prepared charts and enter that data into a grid. Users seemed to sense that this was difficult, and as a result were reluctant to explore very extensively. As a result, we quickly moved to a fully interactive prototype. Despite the implementation headache for us, the interactive prototype allowed us to represent the exploratory nature of Underbelly, which was the most important aspect to evaluate.
This fully interactive prototype was evaluated with four biologists, ranging from masters students in biology to research associates and postdocs. This process was rather informal, and varied somewhat from user to user, depending on their interest level. When users would have a difficult time understanding certain concepts, their thoughts would be documented and then the area of confusion would be explained to them. This allowed us to continue to observe their explorations, which would otherwise be thwarted by misunderstanding.
With a sample size of only four biologists, it is not possible to make claims about the larger population of biologists, however this early study shed some light on what the potential impacts of software like Underbelly might be. None of them spent more than 30 minutes with the software.
The biologists' degree of excitement with the software varied somewhat. An older postdoc who had not done much sequence alignment herself found it to be largely irrelevant to her goals as a biologist. However, two graduate students who had done a significant amount of sequence alignment and genomic search seemed to be genuinely fascinated by Underbelly. They had become somewhat familiar with the idea of the sequence alignment process, but they had resigned themselves to the idea that only computer scientists and bioinformaticists would be allowed to understand how it works. Realizing that they could actually peer into that box that had been permanently labelled "off-limits" seemed to excite them.
Another research associate become somewhat concerned when he saw how simple the sequence alignment algorithm actually is. Before using Underbelly, he seemed to have the conviction that tools like BLAST have an infallible method for determining sequence similarity. When he saw the Underbelly algorithm (which is a simplified version of what is in BLAST), he could see that like all models, it is merely a computational trick and as such it has strengths and weaknesses.
What was surprising was that he started thinking aloud, asking what would happen if you ran the algorithm with one sequence and a second sequence that was exactly the same except that it had the first half swapped with the second half. He felt that kind of swapping was biologically feasible, but he seemed to have an intuition that the algorithm would not handle that situation gracefully. In fact, he was right. The algorithm in Underbelly would give that sequence a rather low score, because only half of the sequence ould be able to match. This is exactly the kind of critical thinking we hoped to encourage.
What remains to be seen is whether biologists would actually invest the time in working with Underbelly. In our user tests, they were asked to explore the software and given support. In some cases, users ran developed misunderstandings that they were unable to overcome. For example, one user developed the conviction that the Maximum Match score was an indicator of similarity between two residues. She could see that this was not the case, but she was unable to figure out why. It is not clear how Underbelly could be improved in this situation.
Underbelly provides learning opportunities to biologists that were previously unavailable, and it lets them do so quickly, with minimal effort. It uses a set of innovative methods for visualizing software in a way that is palletable to users, and has the potential to make a contribution towards solving a serious educational problem in biology.
[1] !BiBiServ (2006) Sequence Alignment Applet http://bibiserv.techfak.uni-bielefeld.de/media/seqanalysis/align-applet.html
[2] Setoft, Peter (1999) Java biosequence alignment applet http://www.dina.kvl.dk/~sestoft/bsa/bsapplet.html
[3] Sumazin, Pavel (2003) Computing Optimal Alignment http://web.cecs.pdx.edu/~ps/CapStone03/dynvis/SimilarityApplet.html
[4] Schleiermacher, Chris (1996) 3D Alignment Visualization http://bibiserv.techfak.uni-bielefeld.de/visualign/
[5] Liu, !PengFei (2004) Global Alignment, Semi-Global Alignment and Local Alignment http://www.cs.yorku.ca/~pfliu/BioAlignApplet/classes/BioAlign.htm
J. Wilson, D. Rosenberg (1988) "Rapid prototyping for user interface design." In Handbook of Human-Computer Interaction, pp. 859-875, M. Helander ed., New York, North-Holland.
Ashburner,M., Ball,C.A., Blake,J.A., Botstein,D., Butler,H., Cherry,J.M., Davis,A.P., Dolinski,K., Dwight,S.S., Eppig,J.T. et al. (2000) Gene Ontology: tool for the unification of biology. Nature Genet., 25, 25–29.
Benson DA, Karsch-Mizrachi I, Lipman DJ, Ostell J, Wheeler DL. (2005) !GenBank. Nucleic Acids Res. Jan 1;33(Database issue):D34-8.
Nicholson, Jeremy K. (2006) Reviewers peering from under a pile of 'omics' data. Nature 440, 992 (20 April 2006)