Most comparisons of wireless ad hoc routing algorithms involve simulated or indoor trial runs, or outdoor runs with only a small number of nodes, potentially leading to an incorrect picture of algorithm performance. In our work, we conducted an outdoor comparison of four different routing algorithms, APRL, AODV, ODMRP, and STARA, running on top of thirty-three 802.11-enabled laptops moving randomly through an athletic field. This comparison provides insight into the behavior of ad hoc routing algorithms at larger real-world scales than have been considered so far. In addition, we compared the outdoor results with both indoor (``tabletop'') and simulation results for the same algorithms, examining the differences between the indoor results and the outdoor reality. We presented these results in a conference paper [GKN+04a] and more extensively in a technical report [GKN+04b].
In addition, we developed a software infrastructure that allowed us to implement the ad hoc routing algorithms and use the same codebase for indoor, outdoor, and simulated trial runs. This approach allowed us to directly validate wireless models against the outdoor experiments described above. The simulator read traces collected from the outdoor experiments, and used them to drive direct-execution implementations of the routing protocols. Because we were able to reproduce the same network conditions as in the real experiment, comparing the routing behavior measured in the real experiment with behavior computed by the simulation, we were able to validate the models of radio behavior upon which protocol behavior depends. We concluded that, contrary to popular belief, it is possible to have fairly accurate results using a simple wireless model (though not as simple as those often used in the literature). We observed, however, that the routing behavior is quite sensitive to one of this model's parameters. The implication is that one should i) use a more complex wireless model that explicitly models point-to-point path loss, ii) use measurements from an environment typical of the one of interest, or iii) study behavior over a range of environments to identify the sensitivities of the protocol's performance under different network conditions. These results were published in a conference paper [LYN+04] and a journal paper [LYN+05].
From our experience implementing, measuring, and simulating ad hoc routing protocols, and from canvassing the literature in the field, we found that many of the assumptions commonly used in most prior research are simply not valid, and that the (often implicit) use of these axioms can lead to questionable results. In a conference paper [KNG+04] and extended technical report [KNE03] we provide a comprehensive review of six assumptions that are still part of many ad hoc network simulation studies, despite increasing awareness of the need to represent more realistic features, including hills, obstacles, link asymmetries, and unpredictable fading. We used our measurements from the above routing experiments to demonstrate the weakness of these assumptions, and showed how these assumptions can cause simulation results to differ significantly from experimental results. We developed a series of recommendations for researchers, whether they develop protocols, analytic models, or simulators for ad hoc wireless networks.
A key player in all of these studies was undergraduate Cal Newport, who spent much of his junior and senior year helping to conduct the experiments and analyze the data. His senior honors thesis [New04] combines many of the results from the papers above. He is now a Ph.D student in computer science at MIT, and is preparing a journal submission based on his thesis.