From KAREN wiki

Waikato Internet Traffic Storage

For researchers studying the way the Internet behaves, there is no substitute for real data. That is, records of the packets passing a point in the Internet. Among the information that can be gleaned from such a record is:

• Which protocols are being used e.g. HTTP for web pages, FTP for some file transfers, BitTorrent for, er, something
• What proportion of users are using which protocols and how much traffic.
• How or badly some services are working by studying the precise times at which packets and their replies are seen
• What unexpected traffic is occurring, often as a result of software fault or misconfiguration
• The way service use and the effects of problems change over time.

Without this kind of information, the design of network and network equipment would involve a large measure of guesswork.

To prevent any personal information from being extracted from traces, all published traces are put through a process of anonymisation. This includes the replacement of all addresses seen with addresses which preserve the identity of the ISP the address came from but cannot be tied to a customer. The contents of packets are removed, leaving only information about the protocol being used.

Since the late 1990's the WAND research group at The University of Waikato, often working with staff at The University of Auckland, have assembled a collection of traces recorded in New Zealand. But meeting requests from researchers overseas or even in New Zealand for terabytes of data has been hard. The usual method to date has been for researchers to ship hard disc drives to Hamilton, for the data to be written to these and the drives then to be shipped back. While the discs themselves have generally stood up well to the stresses of shipping, this process is demanding of staff time.

Why not just make traces available for download? Because data sets running into terabytes can't be transferred over the university's Internet link without a serious effect on other services being used at the time.

Using KAREN, however, is a different matter.

At Our WITS' Beginning

The first of a series of traces to be published is now available for download from the WITS archive. Work is also under way to transfer to Hamilton traffic traces collected by the US National Laboratory for Applied Networking Research, which has now closed down. This is creating an archive of traces from widely differing parts of the Internet over a period of many years.

The archive is available for download only over KAREN, since KAREN is capable of supporting data transfers on this scale without service degradation. This is done by using an IP address allocated by KAREN which can only be reached over KAREN. For reasons including ready availability of that kind of address for IPv6, the WITS trace files are available over IPv6 only.

Before: Research Programmer Perry Lorier prepares data for shipping to the University of Auckland	After: "Lights out" operation

WAND believe this makes WITS New Zealand's first IPv6-only service of any kind at all.

Will requiring researchers to use IPv6 to obtain traces restrict use of the archive? That's not yet clear. But if migration to IPv6 is to happen, people need some encouragement. And if anyone is willing to make the effort to use IPv6, it should be network researchers. It will be interesting to find out.

WITS web site - web pages available over IPv4 or over IPv6 via KAREN only

WITS Archive - FTP archive reachable over IPv4 or IPv6, but trace files accessible only over IPv6 over KAREN.