Network: writes fast / reads slow?

[plugit]

So, I just noticed: copying to the plug from a Windows box, I'm getting about 25-30 megabytes a second. Copying from the plug to the Windows box, I'm getting 2-3 megabytes a second.

Any thoughts?

I've tried two Windows boxes, one Vista and the other Windows 7. I've tried both Samba and SCP. I'm on a gig lan.

OK, I'm going to go Google.

[Pandemonium]

How big are the files you're copying, and are you going to/from the internal NAND or somewhere else?

One explanation could be that files being sent to the plug are being buffered to RAM, allowing the plug to accept data from the network at a higher speed than the NAND can write it.  It then writes it to the NAND as it is able to.  Obviously, it can only read as fast as the NAND will allow, which would explain why it sends data more slowly.

You could test this by moving around some very large files.  If the receiving speed slows down partway through, then you know that this is what is happening.  The buffer fills, forcing the TCP/IP stack to close the receive window which slows down the transmission.

[plugit]

The files are about 500 meg each, and I'm reading to / writing from a fast external drive in a USB enclosure attached to the plug.

CPU utilization seems negligable... I'll test and see what RAM is up to.

My money's on some tweakable sysctl parameter, but I'm not deep in that area, so I'll have to do some reading.

Thanks for the ideas!

[plugit]

Hmmm... So 2.6.17+ has autotuned network settings, eh? Cool. Doesn't help me though.

Oh, and I've changed to 2.6.22.19, no difference.

root@rlyeh:/etc# cat /proc/sys/net/ipv4/tcp_moderate_rcvbuf
1

Yup, it's enabled.

root@rlyeh:/etc# cat /proc/sys/net/ipv4/tcp_rmem
4096    87380   2080768
root@rlyeh:/etc# cat /proc/sys/net/ipv4/tcp_wmem
4096    16384   2080768

...and those values look sane for a box with 512M RAM.

I seem to be at full duplex:

eth0: link up, full duplex, speed 1 Gbps

Hmmm...

[plugit]

You were on to something with the buffering, I think! When reading *from* the plug, my speed starts out fast and drops (quite rapidly) over time.

[plugit]

If someone has the time to try writing a (largeish) file to their plug, and reading it from their plug, from a Windows box, I'd appreciate it. I'm curious to find out if the problem is my setup, or something to be debugged on the plug.

[cbxbiker61]

Can you run iperf between the machines?  I ran iperf last night on a kernel that had debugging enabled and it was running in the range of 200Mbits/sec.  After installing a kernel with debugging disabled (2.6.30-rc4) it's running at 644Mbits/sec.  An Athlon machine runs at 926 Mbits/sec.

644 is acceptable but 200 just didn't seem like it was enough.  I tweeked a lot of ipv4 settings trying to get my speed up and now I realize the settings weren't a problem.  It was just debugging slowing it down.

I did a timed copy of a 50M file to the sheeva and it did about 100MB sec, copying from the sheeva did about 20MB sec.  This of course is to/from a Linux server.  Scp was wayyy slower, the little sheeva just can't keep up with then encryption.

[plugit]

I'll go google iperf now, see if it's something I can get going this evening.

So --- you're seeing similar performance differences, it seems. Five times faster in one direction than the other.

I'd sort of like to figure this one out.

Yeah, I realized after I initalliy posted that scp wasn't going to be a good choice for this test.

[plugit]

Hey, that's a nifty tool. Here we go:

With today's shiny new Windows 7 RC as the client:

C:\Users\Alex\Downloads>iperf.exe -c 192.168.0.3 -t 60
------------------------------------------------------------
Client connecting to 192.168.0.3, TCP port 5001
TCP window size: 8.00 KByte (default)
------------------------------------------------------------
[148] local 192.168.0.2 port 49787 connected with 192.168.0.3 port 5001
[ ID] Interval       Transfer     Bandwidth
[148]  0.0-60.0 sec  2.18 GBytes   312 Mbits/sec

With the plug as the client:

alex@rlyeh:~$ iperf -c 192.168.0.2 -t 60
------------------------------------------------------------
Client connecting to 192.168.0.2, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  3] local 192.168.0.3 port 34301 connected with 192.168.0.2 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-60.0 sec  6.26 GBytes    896 Mbits/sec

[plugit]

And bidirectionally, it's looking similar:

alex@rlyeh:~$ iperf -c 192.168.0.2 -t 60 -d
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 192.168.0.2, TCP port 5001
TCP window size:   277 KByte (default)
------------------------------------------------------------
[  5] local 192.168.0.3 port 49684 connected with 192.168.0.2 port 5001
[  4] local 192.168.0.3 port 5001 connected with 192.168.0.2 port 49831
[ ID] Interval       Transfer     Bandwidth
[  5]  0.0-60.0 sec  3.56 GBytes    509 Mbits/sec
[  4]  0.0-60.0 sec  1.63 GBytes    234 Mbits/sec

[Sutekh]

I would suggest upgrading your kernel to 2.6.30, either rc. The kernel version that is installed by default on the plug only has two schedulers, the default used is anticipatory. Unfortunately anticipatory isn't that good, with the new kernel you can use cfq which is a lot better. I noticed major improvements with accessing my usb disk drive over samba after upgrading.

On top of that you can turn off writing last access times, by mounting with the option 'noatime'.

Reading material:
http://www.redhat.com/magazine/008jun05/features/schedulers/

[cbxbiker61]

My raw iperf rate with sheeva is 624Mbits/sec as client, 576Mbits/sec as server.  So my numbers are a lot more consistent than yours.  I'd probably chock that up to using Linux on both ends.

In the case of my data rate being 100MB to the sheeva and 20MB from the sheeva.  I'm kinda thinking it has to do with using the NFS sync flag on my server.  It seems to me that sync causes the server to block until the write has finished.  I'm not too concerned over it though, since my clients do a lot more reads from the server than writes.

[plugit]

I'll give .30 a spin at some point. I think it's still the debug build out there, isn't it? I'll go see.

[cbxbiker61]

rc4 is not a debug build.  So far it seems to work great.  I put debugging in rc3 due to the NULL referernce errors that were popping up ocassionally.

[plugit]

Uh oh. That means I have to give it a try. Can't help myself, you know how it is. Here we go...