1. After talking to Moritz and Olaf privately and asking them about their
2. nodes, and after running some experiments with some high capacity
3. relays, I've begun to realize that running a fast Tor relay is a
4. pretty black art, with a lot of ad-hoc practice. Only a few people
5. know how to do it, and if you just use Linux and Tor out of the box,
6. your relay will likely underperform on 100Mbit links and above.
7.  
8. In the interest of trying to help grow and distribute the network, my
9. ultimate plan is to try to collect all of this lore, use Science to
10. divine out what actually matters, and then write a more succinct blog
11. post about it.
12.  
13. However, that is a lot of work. It's also not totally necessary to do
14. all this work, when you can get a pretty good setup with a rough
15. superset of all of the ad-hoc voodoo. This post is thus about that
16. voodoo.
17.  
18. Hopefully others will spring forth from the darkness to dump their own
19. voodoo in this thread, as I suspect there is one hell of a lot of it
20. out there, some (much?) of which I don't yet know. Likewise, if any
21. blasphemous heretic wishes to apply Science to this voodoo, they
22. should yell out, "Stand Back, I'm Doing Science!" (at home please, not
23. on this list) and run some experiments to try to eliminate options
24. that are useless to Tor performance. Or cite academic research papers.
25. (But that's not Science, that's computerscience - which is a religion
26. like voodoo, but with cathedrals).
27.  
28. Anyway, on with the draft:
29.  
30.  
31. == Machine Specs ==
32.  
33. First, you want to run your OS in x64 mode because openssl should do
34. crypto faster in 64bit.  
35.  
36. Tor is currently not fully multithreaded, and tends not to benefit
37. beyond 2 cores per process. Even then, the benefit is still marginal
38. beyond just 1 core. 64bit Tor nodes require about one 2Ghz Xeon/Core2
39. core per 100Mbit of capacity.
40.  
41. Thus, to fill an 800Mbit link, you need at least a dual socket, quad
42. core cpu config.  You may be able to squeeze a full gigabit out of one
43. of these machines. As far as I know, no one has ever done this with
44. Tor, on any one machine.
45.  
46. The i7's also just came out in this form factor, and can do
47. hyperthreading (previous models may list 'ht' in cpuinfo, but actually
48. don't support it). This should give you a decent bonus if you set
49. NumCPUs to 2, since ht tends to work better with pure integer math
50. (like crypto). We have not benchmarked this config yet though, but I
51. suspect it should fill a gigabit link fairly easily, possibly
52. approaching 2Gbit.
53.  
54. At full capacity, exit node Tor processes running at this rate consume
55. about 500M of ram. You want to ensure your ram speed is sufficient,
56. but most newish hardware is good. Using on this chart:
57. https://secure.wikimedia.org/wikipedia/en/wiki/List_of_device_bandwidths#Memory_Interconnect.2FRAM_buses
58. you can do the math and see that with a dozen memcpys in each
59. direction, you come out needing DDR2 to be able to push 1Gbit full
60. duplex.
61.  
62. As far as ethernet cards, the Intel e1000e *should* be theoretically
63. good, but they seem to fail at properly irq balancing across multiple
64. CPUs on recent kernels, which can cause you to bottleneck at 100% CPU
65. on one core. At least that has been Moritz's experience. In our
66. experiments, the RTL-8169 works fine (once tweaked, see below).
67.  
68.  
69. == System Tweakscript Wibbles and Config Splatters ==
70.  
71. First, you want to ensure that you run no more than 2 Tor instances
72. per IP. Any more than this and clients will ignore them.
73.  
74. Next, paste the following smattering into the shell (or just read it
75. and make your own script):
76.  
77. # Set the hard limit of open file descriptors really high.
78. # Tor will also potentially run out of ports.
79. ulimit -SHn 65000
80.  
81. # Set the txqueuelen high, to prevent premature drops
82. ifconfig eth0 txqueuelen 20000
83.  
84. # Tell our ethernet card (interrupt found from /proc/interrupts)
85. # to balance its IRQs across one whole CPU socket (4 cpus, mask 0f).
86. # You only want one socket for optimal ISR and buffer caching.
87. #
88. # Note that e1000e does NOT seem to obey this, but RTL-8169 will.
89. echo 0f > /proc/irq/17/smp_affinity
90.  
91. # Make sure you have auxiliary nameservers. I've seen many ISP
92. # nameservers fall over under load from fast tor nodes, both on our
93. # nodes and from scans. Or run caching named and closely monitor it.
94. echo "nameserver 8.8.8.8" >> /etc/resolv.conf
95. echo "nameserver 4.2.2.2" >> /etc/resolv.conf
96.  
97. # Load an amalgam of gigabit-tuning sysctls from:
98. # http://datatag.web.cern.ch/datatag/howto/tcp.html
99. # http://fasterdata.es.net/TCP-tuning/linux.html
100. # http://www.acc.umu.se/~maswan/linux-netperf.txt
101. # http://www.psc.edu/networking/projects/tcptune/#Linux
102. # and elsewhere...
103. # We have no idea which of these are needed yet for our actual use
104. # case, but they do help (especially the nf-contrack ones):
105. sysctl -p << EOF
106. net.ipv4.tcp_rmem = 4096 87380 16777216
107. net.ipv4.tcp_wmem = 4096 65536 16777216
108. net.core.netdev_max_backlog = 2500
109. net.ipv4.tcp_no_metrics_save = 1
110. net.ipv4.tcp_moderate_rcvbuf = 1
111. net.core.rmem_max = 1048575
112. net.core.wmem_max = 1048575
113. net.ipv4.ip_local_port_range = 1025 61000
114. net.ipv4.tcp_synack_retries = 3
115. net.ipv4.tcp_tw_recycle = 1
116. net.ipv4.tcp_max_syn_backlog = 10240
117. net.ipv4.tcp_fin_timeout = 30
118. net.ipv4.tcp_keepalive_time = 1200
119. net.netfilter.nf_conntrack_tcp_timeout_established=7200
120. net.netfilter.nf_conntrack_checksum=0
121. net.netfilter.nf_conntrack_max=131072
122. net.netfilter.nf_conntrack_tcp_timeout_syn_sent=15
123. net.ipv4.tcp_keepalive_time=60
124. net.ipv4.tcp_keepalive_time = 60
125. net.ipv4.tcp_keepalive_intvl = 10
126. net.ipv4.tcp_keepalive_probes = 3
127. net.ipv4.ip_local_port_range = 1025 65530
128. net.core.netdev_max_backlog=300000
129. net.core.somaxconn=20480
130. net.ipv4.tcp_max_tw_buckets=2000000
131. net.ipv4.tcp_timestamps=0
132. vm.min_free_kbytes = 65536
133. net.ipv4.ip_forward=1
134. net.ipv4.tcp_syncookies = 1
135. net.ipv4.tcp_synack_retries = 2
136. net.ipv4.conf.default.forwarding=1
137. net.ipv4.conf.default.proxy_arp = 1
138. net.ipv4.conf.all.rp_filter = 1
139. net.ipv4.conf.default.send_redirects = 1
140. net.ipv4.conf.all.send_redirects = 0
141. EOF
142.  
143. # XXX: ethtool wibbles
144.  
145. # You may also have to tweak some parameters with ethtool, possibly
146. # also enabling some checksum offloading or irq coalescing options to
147. # spare CPU, but for us this hasn't been needed yet.
148.  
149.  
150. == Setting up the Torrc ==
151.  
152. Basically you can just read through the stock example torrc, but there
153. are some as-yet undocumented magic options, and options that need new
154. defaults.
155.  
156. # NumCPUs doesn't provide any benefit beyond 2, and setting it higher
157. # may cause cache misses.
158. NumCPUs 2
159.  
160. # These options have archaic maximums of 2-5Mbyte
161. BandwidthRate 100 MB
162. BandwidthBurst 200 MB
163.  
164.  
165. == Waiting for the Bootstrap and Measurement Process ==
166.  
167. Perhaps the most frustrating part of this setup is how long it takes
168. for you to acquire traffic. If you are starting new at an ISP, I would
169. consider only 200-400Mbit for your first month. Hitting that by the
170. end of the month may be a challenge, mostly because their may be dips
171. and total setbacks along the way.
172.  
173. The slow rampup is primarily due to limitations in Tor's ability to
174. rapidly publish descriptor updates, and to measure relays.
175.  
176. It ends up taking about 2-3 days to hit an observed bandwidth of
177. 2Mbyte/sec per relay, but it can take well over a week or more (Moritz, do
178. you have a better number?) to reach 8-9Mbyte/relay. This is for an
179. Exit node. A middle node will likely gather traffic slower. Also once
180. you crash, you lose it. This bug is about that issue:
181. https://trac.torproject.org/projects/tor/ticket/1863
182.  
183. There is also a potential dip when you get the Guard flag, as our load
184. balancing formulas try to avoid you, but no clients have chosen you
185. yet. Changes to the authority voting on Guards in Tor 0.2.2.15 should
186. make this less drastic. It is also possible that your observed
187. bandwidth will be greater than without it. However, it will still take
188. up to 2 months for clients to choose you as their new Guard.
189.  
190.  
191. == Running temporary auxiliary nodes ==
192.  
193. One way to shortcut this process and avoid paying for bandwidth you
194. don't use is to spin up a bunch of temporary nodes to utilize the CPU
195. and quickly gather that easy first 2MB/sec of observed bandwidth.
196.  
197. But you need the spare IPs to do this.
198.  
199.  
200. == Monitoring ==
201.  
202. Personally, I prefer console-based options like nload, top, and
203. Damian's arm (http://www.atagar.com/arm/) because I don't like the
204. idea of running extra services to publish my monitoring data to the
205. world.
206.  
207. Other people have web-based monitoring using things like munin and
208. mrtg. It would be nice to get a script/howto for that too.
209.  
210.  
211. == Current 1-Box Capacity Record ==
212.  
213. Our setup has topped at 450Mbit, but averages between 300-400Mbit. We
214. are currently having uptime issues due to heat (melting, poorly
215. ventilated harddrives). It is likely that once we resolve this, we
216. will continually increase to our CPU ceiling.
217.  
218. I believe Moritz and Olaf also push this much capacity, possibly a bit
219. more, but with less nodes (4 as opposed to our 8). I hear Jake is also
220. ramping up some Guard nodes (or maybe I didn't? Did I just betray you
221. again Jake?)
222.  
223.  
224. == Did I leave anything out? ==
225.  
226. Well, did I?