This is a log of the work we have done on this project.

Proposal submitted.

Meeting to address project feasibility, delegate tasks, discuss design. AB will work on mining metrics from literature. PK will work on developing client. To meet again on Thursday.

AB - Learned about TCP so I can figure out some ways to measure it.

PK - Got enumeration of adapters running along with capture on selected adapters. Simple GUI to enable / disable adapter capture.

Met briefly - discussed evolving approach. Will meet again on 03/06.

PK - Learned how to create dynamically loaded plug-in interface so that metrics / tests can be self-standing modules.

Met briefly - discussed metrics as known so far. Will meet again Monday.

PK - Wrote a DNSQuery plug-in using low level socket programming so that the query can be sent over non-default interfaces. Plug-in works over default interface (confirmed with WireShark) - have to add another NIC to my computer to test the non-default case.

PK - Added second adapter to desktop. Confirmed plugins are working over alternate adapters. Wrote PING plugin using low level socket interface. This is working nicely. Started traffic driver UI.

AB - Considered reorganizing web pages. Learned to upload files, but held uploading proposal until after reorg. due to possible linking/versioning problems. Researched network metrics from Google search results. Started finding papers from Sigmetrics and related conferences (IMC, MobiCom).

Notes:

• Measure bit error rates (physical and link layers should be able to report)
• Get capacity of links
• Measure achievable bandwidth by saturation

AB, PK - Meeting to discuss what has been learned about metrics so far, demonstrate current client and server functionality. Meet again on Thursday (03/12). AB to complete literature search, learn about SNMP. PK to continue work on client/server.

PK - Server is beginning to take shape. Have architecture mapped out to permit user level and packet level processing to take place in a single application. Server and client have hooked up both within my LAN and from outside to inside my LAN. Server is completely multithreaded and has already handled multiple concurrent clients.

PK - TCPPlugin and server are now happily exchanging traffic.

AB - Went through abstracts of last two years of SIGMETRICS IMC.

AB - Finished literature search of SIGMETRICS IMC. At this point I think I've collected about enough information from the literature. Read about SNMP.

AB - Creating new site organization. The new site should begin here: Multi-Interface Traffic Scheduling. Added temporary version of metrics notes.

PK - After much coding, the traffic driver is fully running including 3 different types of traffic and real time capture at the hardware level. Next step is to add real time capture to the server. Not including experiments, about 2200 lines of code have been written so far.

AB - Worked some on project wiki site. Insufficient permissions yet to reorganize and do what I want.

PK - Created UDP plug-in. Server is sinking them. Next step is still real time capture on the server. Had to modify the TCP plug-in to slow it down. TCP wants to buffer up and send as few packets as possible. I want as many packets as possible so had to add about 100 millisecond delay between sends.

AB - Worked on project wiki site. Created main page, reorganized existing content, uploaded proposal.

PK - Have real time capture on server working. Was a little more difficult than expected as the server doesn't have a complete picture of who is sending to it. It builds a time-based partner dictionary and filters packets based on who is in the dictionary. If no packets have been sent or received with a partner, the partner times out and is removed from the filter dictionary. Next step is to provide a data exchange mechanism between client and server where by one tells the other what it's packet history is.

AB - Looked up useful SNMP variables. Information is on the metrics page.

Met for lunch. Discussed much.

PK - Looked for TCP PUSH support in Windows (.NET). There isn't any. Zilch.

AB - Investigated using C# on Linux. Mono looks feasible. Will try to get it running.

PK - it looks like SNMP won't be usable by us. a) It is hardware dependent - you must know what hardware you're talking to in order to use it b) it requires a known “community name” which works as a password - since we don't know it, we can't use it. :(

PK - modified server to echo any TCP and UDP packets that it sinks. This made the partner stuff I added unnecessary and it has been removed.

AB - Attempt to test database connections with Python. No easy ODBC connectors for MySQL, so will have to do at home where I have more control over my development environment.

AB - Got Mono up and running for C# development on Linux. Connected to Perry's databases. Traffic analysis is next.

AB, PK - Lunch meeting.

AB - Filtered Perry's C# code to remove the “unversionable” bits and committed it to our Subversion repository.

PK - researched and found a means of performing more high resolution timing in .net.

AB - Designed client analysis.

PK - Discovered that the Sprint mobile broadband card is invisible to winpcap. Researched this. Winpcap cannot see modem connections under Vista. Continued researching this on winpcap and sharppcap mailing lists. One fellow suggested bridging the Sprint card to the loop back connector.

I was in the process of doing this (see above) when I could not reinstall the Sprint drivers. Looking for a new set of drivers, I found a completely revamped set from Sprint. Installing, and guess what - the NDIS adapter can not be directly seen by winpcap. Successfully sending and receiving via our software over the Sprint card to our server.

Learned some fascinating things about Windows networking. The metric you specify in the network settings doesn't do diddly. If you really want to change network metrics you have explicitly set metrics in the routing tables. This is done by changing your default route for each adapter to the same thing with a new metric specified. THIS works. This was why my gigabit link would always take 2nd place to a 2 megabit link. The 2 megabit link showed up in the routine table with a metric of 80 while the gigabit link had a metric of 266! None of these numbers corresponded to the network metrics I specified in the adapter options.

I wrote a large TCP block sender plug-in. This caused a lot of opportunity to reassess the capture code. And, finally, caused a lot of pouring over wireshark captures comparing them to mine. This caused some learning about D-Link NIC options.

I was seeing TCP packet with lengths near 64KB on the sending side (my machine) in the raw capture. This couldn't be says I. The receiving side (through the Internet) saw only normal Ethernet sized packets. There were a lot of lost segments and retires as the sender insisted it sent 64K and the receiver insisted it received only 1460 bytes. So pouring over wireshark I went.

Get this, in order to get fast performance with my NAS I enabled Jumbo packets (9KB). This is OK - as the NAS and my machine (and my switches) all understand Jumbo packets. The router doesn't understand Jumbo packets so the negotiation between my machine and the router is SUPPOSED to drop packet sizes back to normal for that path. Turned out I had some other option on - Large Packet Offload. This is REALLY how large packets happen. But THIS OPTION doesn't get negotiated. So, when I blast something out to the router to the Internet, I get huge numbers of retries because I'm sending huge segments with the Don't Fragment flag set.

So now I understand and can reconcile wireshark to my server and client logs. But I can't get optimum performance to my NAS AND the Internet at the same time. Spent 7 hours on this all today.

At this point I think the software is done to the point where meaningful capture is all there. Crud. I still have to send capture data from the server to the client when asked. I'll do this next.

Still coding (later that day). Added “runid” to the UDP and both TCP plug-ins. The TCP case involved added a preamble (first bytes sent over the socket) so that both sides can have the same runid. I've had enough for today.

AB - Continued work on basic analysis. Unfortunately not yet ready to commit. Complications, bleary eyes, hazy mind, the usual suspects for excuses.

PK - Rewrite of the client UI bringing all plugins and all adapters under a single executable. Very pleased. Captured data on 3 adapters X 5 plugins simultaneously (15 data streams).

AB - Set up Mono and MySQL access from Mono.

PK - Writing HTTP plugin. Tweaks of capture code to lock resources less often and in shorted intervals.

AB - SQL data analysis

AB, PK - Meeting to discuss progress and issues. We are slightly behind schedule. AB's fault.

PK - HTTP plugin giving me more trouble than it's worth. Recoded to hit a single web site per major interval so as to allow finer grained capture filters.

PK - A lot of data capture tests - installed and run on laptop.

AB - Tried to dump project DBs. Decided default number of plugin subsamples. Ping analysis.

AB, PK - Meeting. Status and issues. ICMP packet lengths confusing (not adding up).

AB - DNS analysis.

PK - Finally have HTTP plugin where it should be. Added additional capture fields as per Aubrey.

PK - propagated code to correctly extract payloads without control characters.

AB - Finished client download of server data.

AB, PK - Meeting.

AB, PK - Meeting with Suman, who advises altering project direction. Thereafter, meeting with ourselves to discuss and start on new project direction. We will focus exclusively on scheduling data to multiple TCP streams from now on. Discussed necessary architecture, priorities, collaboration points.

AB - Prepare presentation for meeting with Suman.