ot Vladsun(19-10-2006)

reiting (20)   [ dobre ]  [ zle ]

Printer Friendly Variant za otpechatvane

Optimizatsiia na iptables i tc pravila.

TSelta na statiiata e da pokazhe nachini za podobriavane efektivnostta i rabotata na ruteri, koito izvurshvat i trafik kontrol vurhu potrebitelite. Tova e postignato chrez optimizirane na pravilata izpolzvani v iptables i tc instrumentite v rutera. Kato dopulnitelno chetivo kum tazi statiia e statiiata na Uvigii "Razpredeliane na trafik ot dva (ili poveche) internet dostavchika".

1. Predvaritelna podgotovka
Statiiata NE e prednaznachena za nachinaeshti v tazi oblast i za tiah e nalozhitelno da razgledat niakoi dokumentatsii:



2. Sistemni iziskvaniia
Predvaritelno triabva da e instaliran (pachnat) sledniia softuer:
  • v iadroto - QoS (htb, sfq);
  • iptables;
  • iptables - IPMARK;
  • tc;
  • Perl (za skriptovete izpolzvani v tazi statiia).


3. Problemut
Nai-chesto sreshtaniia algoritum za razpredeliane na trafik mezhdu potrebiteli e sledniia:
  1. razdeliane na trafika na vhodiasht i izhodiasht;
  2. razdeliane na trafika na bulgarski i mezhdunaroden chrez podhodiashto markirane na paketite (vzh. "Razpredeliane na trafik ot dva (ili poveche) internet dostavchika");
  3. razdeliane na trafika za vseki potrebitel chrez povtorno markirane na paketite za vseki vid trafik;
Osven tova obiknoveno se nalaga da se pravi kontrol na dostupa na potrebitelite - chrez izpolzvaneto na iptables pravila v PREROUTING ili FORWARD verigite.
Problemut pri povecheto ot taka realizirani algoritmi e znachitelnoto natovarvane na survura poradi niskata efektivnost na nabora ot izpolzvanite pravila v iptables i tc. Za po-detailno obiasnenie shte pokazha primeren nabor ot pravila izpolzvani v iptables. Predpolaga se, che razdelianeto na trafika na bulgarski i mezhdunaroden, suotvetno vhodiasht/izhodiasht veche e napraven i sa polucheni slednite potrebitelski definirani verigi - BG_IN, BG_OUT, INT_IN, INT_OUT. V tezi verigi triabva da se izvurshi markiraneto za vseki potrebitel, t.e. dobaviame po 4 pravila na potrebitel:
Primeren kod 
iptables -A BG_OUT -s IP1 -j MARK --set-mark MARK_IP1_BG_OUT
 iptables -A INT_IN -d IP1 -j MARK --set-mark MARK_IP1_INT_IN
 iptables -A INT_OUT -s IP1 -j MARK --set-mark MARK_IP1_INT_OUT
Malkata efektivnost na tozi metod e porodena ot lineinoto (posledovatelnoto) obhozhdane na tezi pravila za vseki paket. Po tozi nachin vseki paket minava prez okolo 260 pravila, pri tova samo v tezi verigi! Oshte po-loshiiat variant, pri koito niama razdelian na izhodiasht/vhodiasht trafik, broiat na pravilata obhodeni ot vseki paket e nad 510! Pri tozi variant gornite pravila izglezhdat po tozi nachin:
Primeren kod 
iptables -A BG -s IP1 -j MARK --set-mark MARK_IP1_BG_OUT
 iptables -A INT -d IP1 -j MARK --set-mark MARK_IP1_INT_IN
 iptables -A INT -s IP1 -j MARK --set-mark MARK_IP1_INT_OUT
Sled markiraneto na paketite e neobhodimo sushto taka da se suzdadat tc pravilata, koito shte opredeliat minimalnata i maksimalnata skorost za vseki edin potrebitel v zavisimost ot vida na trafika. Tova oznachava, che za vseki potrebitel shte imame po 4 klasa, 4 qdisc (b.a. - daite svesten prevod) i 4 filtura - po 2 za vseki interfeis. Priakoto sledstvie ot tozi nachin na postroiavane na tc pravilata e otnovo lineinoto obhozhdane na tc filtrite. V sluchaia ne znam dali pri dostigane na suotvetniia filtur za markiraniia paket proslediavaneto na paketa v sledvashtite filtri produlzhava ili ne. No dori i v nai-dobriia sluchai imame srednostatisticheski broi na obhodenite filtri - 1/2 ot vsichki filtri, ili 254 filtura. Tova sushto e svurzano s dopulnitelno (pri tova izlishno) natovarvane na rutera.

4. Reshenieto
SHTe pokazha reshenie (ne tvurdia, che e edinstvenoto ili nai-dobroto), pri koeto problemut s lineinoto i izlishno obhozhdane na pravila v iptables i suotvetnite filtri v tc e izbiagnat napulno. Vsushtnost samoto reshenie e realizirano otdavna, no za suzhalenie ne e dobre dokumentirano (da ne kazvam izobshto). Neka razgledame man-a na iptables i po-spetsialno IPMARK: 
  IPMARK
        Allows you to mark a received packet basing on its IP address. This can replace many mangle/mark entries with only one,
        if you use firewall based classifier.
 
        This target is to be used inside the mangle table, in the PREROUTING, POSTROUTING or FORWARD hooks.
 
        --addr src/dst
               Use source or destination IP address.
 
        --and-mask mask
               Perform bitwise `and' on the IP address and this mask.
 
        --or-mask mask
               Perform bitwise `or' on the IP address and this mask.
 
        The order of IP address bytes is reversed to meet "human order of bytes": 192.168.0.1 is 0xc0a80001. At first the `and'
        operation is performed, then `or'.
 
        Examples:
 
        We create a queue for each user, the queue number is adequate to the IP address of the user, e.g.:  all  packets  going
        to/from 192.168.5.2 are directed to 1:0502 queue, 192.168.5.12 -> 1:050c etc.
 
        We have one classifier rule:
 
               tc filter add dev eth3 parent 1:0 protocol ip fw
 
        Earlier we had many rules just like below:
 
               iptables -t mangle -A POSTROUTING -o eth3 -d 192.168.5.2 -j MARK --set-mark 0x10502
 
               iptables -t mangle -A POSTROUTING -o eth3 -d 192.168.5.3 -j MARK --set-mark 0x10503
 
        Using IPMARK target we can replace all the mangle/mark rules with only one:
 
               iptables -t mangle -A POSTROUTING -o eth3 -j IPMARK --addr=dst --and-mask=0xffff --or-mask=0x10000
 
        On the routers with hundreds of users there should be significant load decrease (e.g. twice).
Kato 'zhoker' vi davam i dopulnitelna informatsiia, koiato otkrih v Internet (sluchaino): 
 Something I've only just noticed from a comment in the code - htb can use mark without the need for lots of filters. 
 You only need one empty filter on the root (maybe you can still nest) like -
 
 tc filter add dev eth0 parent 1:0 protocol ip prio 1 fw
 
 and then if you arrange for your classes to be the same minor numbers as the marks it will behave like using classify. 
 You need to set the major number of your htb (1 in example above) in the top 16 bits of the mark. 
 There is also a netfilter pom-ng patch IPMARK that will set marks based on ipaddress. 
 
 Andy.
Neka sega preveda na bulgarski i malko po-podrobno vsichko tova:
  • IPMARK: markira paketite, kato za stoinostta na markera izpolzva stoinostta na samoto IP;
  • IPMARK: priema kato parametri 3 neshta - IP-to po koeto shte markirame i 2 parametura za upravlenie na polucheniia MARK;
  • IPMARK: IP-to po koeto she markirame se vzima direktno ot paketa i v zavisimost ot tova dali --addr parametura e src ili dst, se vzima suotvetno src IP-to ili dst IP-to;
  • IPMARK: 2-ta parametura za upravlenie na polucheniia MARK realizirat pobitovo I i ILI vurhu stoinostta na IP adresa;
Primer: iskame da markirame vsichki IP-ta ot mrezha 192.168.0.0/24, taka che markerut da e poslednata grupa ot IP-to sumirano s 512. Pri tova polozhenie AND-maskata e 0x00ff - vzimame poslednite 2 shestnaisetichni chisla i chrez OR-maska 0x0200 (512 dec.) sumirame s 512. Taka poluchenite paketi shte imat slednite markeri: 
 	192.168.0.1   - MARK = 0x201;
 	192.168.0.2   - MARK = 0x202;
 	192.168.0.3   - MARK = 0x203;
 	...........
 	...........
 	192.168.0.253 - MARK = 0x2FD;
 	192.168.0.254 - MARK = 0x2FE;
Vuv vtorata dadena po-gore informatsiia se obiasniava, che ako ostavim tc filtur bez parametri za fw poleto, to togava tozi filtur deistva direktno po MARK poleto na paketa. Deistvieto na filtura e slednoto:
  • razdelia MARK poleto na 2 chast - mladsha tetrada i starsha tetrada;
  • mladshata tetrada opredelia minor nomera na klasa, a starshata - major nomera na klasa.
Ako za primer vzemem paket markiran s 0x0001 0005, tozi paket se nasochva kum klas sus classid = 1:5.

Neka sega sled tezi raziasneniia da poluchim reshenie za razpredeliane na trafika na edna S-klas mrezha (prim. 192.168.0.0):
Primeren kod 
iptables -A BG_IN -d 192.168.0.0/24 -j IPMARK --addr=dst --and-mask=0xff --or-mask=0x10100
 iptables -A BG_OUT -s 192.168.0.0/24 -j IPMARK --addr=dst --and-mask=0xff --or-mask=0x10200
 iptables -A INT_IN -d 192.168.0.0/24 -j IPMARK --addr=dst --and-mask=0xff --or-mask=0x10300
 iptables -A INT_OUT -s 192.168.0.0/24 -j IPMARK --addr=dst --and-mask=0xff --or-mask=0x10400
Dobaviame obshtite filtri (definiraneto na tc pravilata za obshtiia, bulgarskiia i mezhdunarodniia trafik, suotvetno vhodiasht/izhodiasht niama da bude razglezhdan v tazi statiia):
Primeren kod 
tc filter add dev eth0 parent 1:0 protocol ip prio 1 fw
 tc filter add dev eth1 parent 1:0 protocol ip prio 1 fw
i edin Perl skript za dobaviane na edno IP (poslednite 3 tsifri):
ip.add 
#!/usr/bin/perl
 ($ip, $bgmin, $bgmax, $intmin, $intmax) = @ARGV;
 $id = sprintf("%X", $ip + 0x200);
 $class_bg_ul = "tc class add dev eth0 parent 1:15 classid 1:0".$id." htb rate ".$bgmin."Kbit ceil ".$bgmax."Kbit prio 5";
 $qdisc_bg_ul = "tc qdisc add dev eth0 parent 1:0".$id." handle 0".$id." sfq perturb 10 ";
 
 $id = sprintf("%X", $ip + 0x100);
 $class_bg_dl = "tc class add dev eth1 parent 1:10 classid 1:0".$id." htb rate ".$bgmin."Kbit ceil ".$bgmax."Kbit prio 5";
 $qdisc_bg_dl = "tc qdisc add dev eth1 parent 1:0".$id." handle ".$id." sfq perturb 10 ";
 
 $id = sprintf("%X", $ip + 0x400);
 $class_int_ul = "tc class add dev eth0 parent 1:25 classid 1:0".$id." htb rate ".$intmin."Kbit ceil ".$intmax."Kbit prio 4";
 $qdisc_int_ul = "tc qdisc add dev eth0 parent 1:0".$id." handle ".$id." sfq perturb 10 ";
 
 $id = sprintf("%X", $ip + 0x300);
 $class_int_dl = "tc class add dev eth1 parent 1:20 classid 1:0".$id." htb rate ".$intmin."Kbit ceil ".$intmax."Kbit prio 4";
 $qdisc_int_dl = "tc qdisc add dev eth1 parent 1:0".$id." handle ".$id." sfq perturb 10 ";
 
 
 `$class_bg_ul`;
 `$qdisc_bg_ul`;
 
 `$class_int_ul`;
 `$qdisc_int_ul`;
 
 `$class_bg_dl`;
 `$qdisc_bg_dl`;
 
 `$class_int_dl`;
 `$qdisc_int_dl`;
Po tozi nachin sukratihme 4*254=1016 pravila v iptables na 4!, i oshte 1016 filtura v tc sushto na 4!!! :)

5. Dopulnitelna optimizatsiia
Po-gore beshe otbeliazano, che chesto se nalaga izpolzvaneto na iptables pravila za realiziraneto na kontrol vurhu dsotupa na potrebitelite. Nai-chesto realizatsiiata e po sledniia nachin:
Primeren kod 
 iptables -A FORWARD -s IP1 -j ACCEPT
 iptables -A FORWARD -d IP1 -j ACCEPT
 ....................................
 ....................................
 iptables -A FORWARD -s IPn -j ACCEPT
 iptables -A FORWARD -d IPn -j ACCEPT
Ili kazano po-drug nachin: vsichki potrebiteli, koito imat razreshenie za dostup do Internet izrichno se dobaviat vuv FORWARD verigata s ACCEPT. Na ostanalite im se otkazva dostup zaradi izbranata politika na tazi veriga - DROP. Otnovo imame lineino obhozhdane na mnogo pravila, koeto obache izglezhda absolyutno nalozhitelno. Tozi put shte prilozha druga strategiia - shte transformiram tezi lineino razpolozheni pravila v durvovidna struktura, chrez posledovatelno razdeliana na 2. Taka pri obhozhdaneto na pravilata za edin paket shte se realizira dvoichno tursene, koeto kakto se znae e nai-efektivnoto za naredeni danni (popravete me, ako gresha). Za da se iziasnia shte pokazha nagledno naborut ot pravila v poluchenoto durvo za vhodiasht bulgarski trafik pri mrezha 192.168.2.0/24 :
Primeren kod 
Purva iteratsiia:
 iptables -t filter -N BG_IN_192_168_2_0-255
 iptables -t filter -A FORWARD_IN -d 192.168.2.0/24 -j BG_IN_192_168_2_0-255
 iptables -t filter -N BG_IN_192_168_2_0-127
 iptables -t filter -A BG_IN_192_168_2_0-255 -d 192.168.2.0/25 -j BG_IN_192_168_2_0-127
 iptables -t filter -N BG_IN_192_168_2_128-255
 iptables -t filter -A BG_IN_192_168_2_0-255 -d 192.168.2.128/25 -j BG_IN_192_168_2_128-255
 
 Vtora iteratsiia:
 iptables -t filter -N BG_IN_192_168_2_0-255
 iptables -t filter -A FORWARD_IN -d 192.168.2.0/24 -j BG_IN_192_168_2_0-255
 iptables -t filter -N BG_IN_192_168_2_0-127
 iptables -t filter -A BG_IN_192_168_2_0-255 -d 192.168.2.0/25 -j BG_IN_192_168_2_0-127
 iptables -t filter -N BG_IN_192_168_2_0-63
 iptables -t filter -A BG_IN_192_168_2_0-127 -d 192.168.2.0/26 -j BG_IN_192_168_2_0-63
 iptables -t filter -N BG_IN_192_168_2_64-127
 iptables -t filter -A BG_IN_192_168_2_0-127 -d 192.168.2.64/26 -j BG_IN_192_168_2_64-127
 iptables -t filter -N BG_IN_192_168_2_128-255
 iptables -t filter -A BG_IN_192_168_2_0-255 -d 192.168.2.128/25 -j BG_IN_192_168_2_128-255
 iptables -t filter -N BG_IN_192_168_2_128-191
 iptables -t filter -A BG_IN_192_168_2_128-255 -d 192.168.2.128/26 -j BG_IN_192_168_2_128-191
 iptables -t filter -N BG_IN_192_168_2_192-255
 iptables -t filter -A BG_IN_192_168_2_128-255 -d 192.168.2.192/26 -j BG_IN_192_168_2_192-255
 
 ...................
i taka dokato poluchim interval sudurzhasht izbraniia ot nas broi IP-ta, koito shte budat obhodeni posledovatelno (prim. 2 ili 4). Po tozi nachin srednostatichsticheskiia broi na pravilata, koito triabva da premine vseki paket e broi_deleniia puti po-maluk ot inache neobhodimiia (127).
Tui kato tazi zadacha e dosta trudoemka prilagam Perl skript (mozhe bi ne nai-dobriia) za reshavaneto i:
rc.netsubdiv 
#!/usr/bin/perl
 
 # Mrezha/24
 $net            = "192.168.2.0";
 
 # Tablitsata v iptables
 $table          = "filter";
 
 # Verigata, ot koiato trugvame
 $start_chain    = "FORWARD";
 
 # Prefiksa na verigite, koito poluchavame v durvoto
 $chain          = "BG";
 
 # Da se slagat li pravila za src/dst
 $src            = 1;
 $dst            = 1;
 
 # Do koga da se deli (min=1, maks=256) (vinagi stepen na 2)
 $subdiv         = 128;
 
 # Qsno ;)
 $ipt           = "/usr/local/sbin/iptables";
 
 sub divide_net
 {
         my($net,$maxdiv, $div, $subnet, $prev_chain, $int_min) = @_;
         my $subint = 256/$div;
         my $ip;
         
         if ($div > $maxdiv)
         {
                 return;
         }
         
         $net2 = $net;
         $net2 =~ s/\./_/g;
         
         for ($ip=0; $ip<$div and $ip<2 ; $ip++)
         {
                 $min = $ip*$subint+$int_min;
                 $max = $min + $subint - 1;
                 
                 print $ipt." -t ".$table." -N "
                 .$chain."_IN_".$net2."_".$min."-"
                 .$max."\n";
                 
                 if ($div > 1)
                 {
                         if ($dst == 1)
                         {
                                 print $ipt." -t ".$table." -A "
                                 .$chain."_IN_".$prev_chain." –d "
                                 .$net.".".$min."/".$subnet." -j "           
                                 .$chain."_IN_".$net2."_".$min."-".$max."\n";
                         }
                         if ($src == 1)
                         {
                                 print $ipt." -t ".$table." -A "
                                 .$chain."_OUT_".$prev_chain." -s "
                                 .$net.".".$min."/".$subnet." -j "
                                 .$chain."_OUT_".$net2."_".$min."-".$max."\n\n";
                         }
                 }
                 else
                 {
                         if ($dst == 1)
                         {
                                 print $ipt." -t ".$table." -A ".$start_chain."_IN -d "
                                 .$net.".0/".$subnet." -j ".$chain."_IN_"
                                 .$net2."_".$min."-".$max."\n";
                         }
                         if ($src == 1)
                         {
                                 print $ipt." -t ".$table." -A ".$start_chain."_OUT -s "
                                 .$net.".0/".$subnet." -j ".$chain."_OUT_"
                                 .$net2."_".$min."-".$max."\n\n";
                         }
                 }
                 
                 $prev_chain2 = $net2."_".$min."-".$max;
                 divide_net($net, $maxdiv, $div*2, $subnet+1, $prev_chain2, $min);
                 
         }
 }
 
 $net =~ /^((\d+)\.(\d+)\.(\d+))/;
 $net = $1;
 divide_net($net, $subdiv, 1, 24, '', 0);
Promenlivite v nachalato na skripta se izpolzvat konfigurirane. Estestveno za spetsifichni nuzhdi tozi skript shte se nalozhi da se modifitsira. Zabelezhete, che samiia skript ne izpulniava iptables pravilata. Neobhodimo e izhodut ot tozi skript da se prenasochi kum fail i veche ot tozi fail da se izpulniat iptables komandite.
Sled kato sme postroili durvoto e nuzhno da dobavim pravilata za kontrol na dostupa za vsiako IP. Otnovo na pomosht idva Perl:
Primeren kod 
sub get_subnet_interval()
 {
         ($ip, $subdiv) = $_;
         
         my $subint = 256/$subdiv;
         my $net = $ip;
         
         $net =~ /^((\d+).(\d+).(\d+))/;
         $net = $1;
         $ip  =~ /^(\d+).(\d+).(\d+).(\d+)/;
         $ip = $5;
         
         $min = (int($ip/$subint))*$subint;
         $max = $min + $subint - 1;
         
         return ($min."-".$max);
 }
CHrez tazi funktsiia poluchavate verigata, v koiato se namira dadenoto ot Vas IP. Funktsiiata vrushta string, koito e ot vida polzvan v predishniia skript za generirane na durvoto, no bez prefiksite na verigata. Tova e napraveno s tsel po-universalnata upotreba na funktsiiata.
E, tova e :) Ochakvam komentari!


<< Delegirane na prava chrez /etc/sudoers | Plavno preminavane ot edna OS/Distributsiia na druga >>