Blog Rss FeedOpening a file descriptor in shell can be useful for two things: manipulating several input and output as the same time, and for performance.
How to create a file descriptors
exec 6</tmp/foo, opens the file /tmp/foo for input on the file descriptor #6. This is equivalent to the system call open("/tmp/foo", O_RDONLY)
exec 7>/tmp/bar, opens the file /tmp/bar for output on the file descriptor #7. If the file already exist it is recreated. This is equivalent to the system call open("/tmp/bar", O_WRONLY|O_CREAT).
exec 7>>/tmp/bar, opens the file /tmp/bar for appending on the file descriptor #7. If the file does not exist it is created. This is equivalent to the system call open("/tmp/bar", O_APPEND|O_CREAT).
exec 6&- close the file descriptor.
You can also do other file descriptor manipulation such as duplicating file descriptors, or redirecting standard IO to files. To learn about it I invite you to read the sh, ksh or bash man page.
Performance
When your shell script evaluates a line like "echo $x >>/tmp/file", the file is opened, the content of the variable is written and the file is then closed. If this is done a couple of times in your shell script, that's fine. But if you have a loop with several thousand writes, using a file descriptor can dramatically improve the performace of your script.
Here is a quick test I have done with the following shell script.
#!/bin/bash cat /tmp/foo | while read a do echo $a >>/tmp/bar done
The input file is a file containing 2.000.000 lines. This means that the shell script is going to open, write, and close /tmp/bar 2 million times. After running the shell, I get the following results.
$ time ./bar.sh real 18m26.416s user 4m9.820s sys 8m42.876s
Now using the same shell script with file descriptors.
#!/bin/bash # open the files exec 6</tmp/foo exec 7>/tmp/bar # data "processing" cat <&6 | while read a do echo $a >&7 done
I get these results. As you can see we have a performance increase of almost 4.5.
$ time ./bar.sh real 4m30.501s user 2m22.004s sys 1m17.993s
Playing with several file descriptors
Here is a small example of a shell script using several file descriptors. This script reads a file with the following columns "ip_address trafic_in trafic_out", and writes two files trafic_in and trafic_out.
#!/bin/bash # open the two output files exec 6>/tmp/trafic_in.dat exec 7>/tmp/trafic_out.dat #open the file containing the data for input. exec 8</tmp/all_trafic.dat # data processing grep -v '^#' <&8 | while read line do set - $(echo $line) echo "${1} ${2}" >&6 echo "${1} ${3}" >&7 done #close the file descriptors exec 6<&- exec 7<&-
When executed this code produces the following output
$ head -4 /tmp/all_trafic.dat # Host In (bytes) Out (bytes) Total (bytes) 172.16.1.1 2728242803 17456323158 20184565961 172.16.1.2 62238068877 146358768518 208596837395 172.16.1.3 123056619706 150682371892 273738991598 $ ./split_trafic.sh $ $ head -4 /tmp/trafic_in.dat 172.16.1.1 2728242803 172.16.1.2 62238068877 172.16.1.3 123056619706 172.16.1.4 7684221078 $ head -4 /tmp/trafic_out.dat 172.16.1.1 17456323158 172.16.1.2 146358768518 172.16.1.3 150682371892 172.16.1.4 1931647367
Comments
Posted by: najaf husain zaidi Oct 11, 2008 @ 02:41
great article . thaks a lot najaf husain zaidi
Posted by: Pork Chops Apr 06, 2009 @ 18:58
Is there a way to use a variable for the file descriptor in a script instead of "hard coding" it? For example:
exec 8</tmp/all_trafic.dat # works
INFILE="/tmp/all_trafic.dat" exec 8<${INFILE} # works
FD=8 INFILE="/tmp/all_trafic.dat" exec ${FD}<${INFILE} # no worky -bash: exec: 8: not found
Posted by: Allan May 04, 2009 @ 00:32
Pork Chops: You will need to use:
eval "exec ${FD}<${INFILE}"
We want variables expanded before fd dup’ing, so we quote it to expand variables but not do fd dup’ing and re-run the (expanded) string via ‘eval’.
Posted by: Gapato Aug 17, 2009 @ 05:58
Thanks :-)
Posted by: Education support forum Nov 12, 2009 @ 22:23
This script reads a file with the following columns "ip_address trafic_in trafic_out", and writes two files trafic_in and trafic_out.
Posted by: chat Nov 23, 2009 @ 03:39
After half an hour of Google-ing and applying a little horse-sense, I discovered (from counting the lines in a Windows GUI text editor) that the blank line was at the very end, right after the last perc ("%") delimiter. Pico, as has been my experience, loves adding those blanks. (Anticipating comments on the order of "Why not use [insert favorite shell editor here]": I can say that after the third time of not being able to quit out of vim or emacs in OS X's Terminal app, pico seemed like a breath of fresh air; for all it's attendant quirks and foibles, I still prefer it to the others.)
Posted by: sohbet Nov 23, 2009 @ 03:46
thakns
Posted by: cheap Christian Louboutin Boots Nov 25, 2009 @ 21:41
[HTML_REMOVED]cheap Chloe[HTML_REMOVED] [HTML_REMOVED]cheap Christian Louboutin[HTML_REMOVED] [HTML_REMOVED]cheap Christian Louboutin Boots[HTML_REMOVED]
Posted by: Chloe Nov 25, 2009 @ 21:42
http://www.christianlouboutincheap.net/Chloe/
Posted by: free online games Nov 27, 2009 @ 00:42
Thanks :-)
Posted by: gmt master ii Dec 08, 2009 @ 00:52
Nice one! Thanks for sharing ! High quality Replica Rolex gmt master ii,welcome to choose! Thans!
Posted by: SEO Experts Dec 15, 2009 @ 16:23
Thank you very much man! It's a new platform actually for me. I've only zero experience with shell coding. Never tried that. While checking with your archive I saw that and just got inside thinking to learning some thing new.
:) thanks
Posted by: sell ugg ugg boots online ugg boots Dec 16, 2009 @ 19:00
1 Chestnut UGG Bailey Button Boots
2 Rose UGG Classic Cardy Boots 3 Sand UGG Classic Mini Boots 4 UGG Classic Argyle Knit whit 5879 5 UGG 5822 Stripe Cable Knit Chocolate White 6 Chestunt Genuine Australia classic Tall 7 Sand UGG Ultra Shotr Boots 8 Chestnut UGG Sundance Boots
Posted by: links of london Dec 20, 2009 @ 22:58
Early morning time, the mist is dim. Curved river bank, willow obviously elegant like smoke.In a hurry the pedestrian, the moist hair clothing which fills by the mist. Most beautiful, when is in the fog the park, pavilion Taiwan pavilion, porch bridge park, fuzzy, partly visible. Let the human think that imitates, if places oneself in unreal, is considering, is in the world, plays in the heavenly palace?
Posted by: oyun indir full oyun program indir full program Dec 23, 2009 @ 16:47
Usually people don't know or have forgotten that they can open file descriptors in shell. +1 :)
Good Job m8. :)
Posted by: deliver flowers japan Dec 29, 2009 @ 03:56
Early morning time, the mist is dim. Curved river bank, willow obviously elegant like smoke.In a hurry the pedestrian, the moist hair clothing which fills by the mist. Most beautiful, when is in the fog the park, pavilion Taiwan pavilion, porch bridge park, fuzzy, partly visible. Let the human think that imitates, if places oneself in unreal, is considering, is in the world, plays in the heavenly palace?
Posted by: replica watches Jan 07, 2010 @ 21:54
The tag heuer rubber strap rolex doesn’t stray from rolex watches the strap pattern. Even replica watches though breitling watches the strap patek philippe patterns of audemars piguet watches from this brand chanel watches are familiar to us, it approaches a very sport and innovative montblanc watches appearance. This brand fake watches are always trying new replica watches watches with rolex watches new designs. This New replica rolex Concord C1 Chronograph tag heuer Watch can best embody this feature.This time Concord breitling watches offers us
Posted by: replica watches Jan 11, 2010 @ 18:43
As the business grows, replica rolex paypal has also jumped into an international brand. It is worth mentioning that, rolex replica is the ancestor of today's brand-oriented, in order to protect the quality and brand name will be printed on their products, the history of fashion in the world, is the first one first. replica watch paypal,replica watches,fake rolex.
Posted by: film izle Jan 17, 2010 @ 04:37
Thanks for your good website and for sharing your experiences.
Posted by: divx film izle Jan 17, 2010 @ 04:38
Thank you for sharing your friends. Hope to see you another day.
Posted by: Wholesale Laptops Jan 17, 2010 @ 22:06
As the business grows, replica rolex paypal has also jumped into an international brand
Posted by: links of london Jan 19, 2010 @ 00:26
This brand fake watches are always trying new replica watches watches with rolex watches new designs.
Posted by: film izle Feb 09, 2010 @ 16:47
Thanks for document...
Posted by: şentürk inşaat Feb 09, 2010 @ 16:48
good documents, thanks
Posted by: ahmad Mar 03, 2010 @ 05:20
thanks for the document, i need to know what is the difference between [N]<&WORD and [N]>&WORD as in the info bash.
It will be helpful if anyone could include a graph for file descriptor before and after different command execution.
Posted by: Nike Air Max Mar 04, 2010 @ 00:56
So beautiful sharing!Thank you very much.
Posted by: tag heuer watch Mar 04, 2010 @ 23:29
can show a man not only his wealthy but also his high taste.tag heuer watch As the society developed, many people are pay more attention to one's wearing than before, especially a watch. tag heuer watch If you are wearing a famous brand watch, people will look up you, also will love to do business with you. You will be a successful man with a such luxury fashion watch.
Posted by: Nike Air Max Mar 11, 2010 @ 19:15
Best wishes!Your blog is very good!
Posted by: omega moon watch Mar 11, 2010 @ 20:42
I recently came across your blog and have been reading along. I thought I would leave my first comment. I don't know what to say except that I have enjoyed reading. Nice blog. I will keep visiting this blog very often.
Posted by: Nike Air Max Mar 18, 2010 @ 02:48
It is so nice.
Posted by: replica handbags Apr 04, 2010 @ 09:30
A very good article, I will always come in.
Posted by: VillarrealELOISE24 Apr 05, 2010 @ 21:54
It's cool that we can receive the [HTML_REMOVED]credit loans[HTML_REMOVED] and it opens new chances.
Posted by: buy essays Apr 08, 2010 @ 19:40
To research the this good post is not really easy and really good solution was to buy essay at the term paper writing service.
Posted by: dissertation writing Apr 12, 2010 @ 16:38
Oh finally, I have got good information reffering to this topic? I offer to find the buy thesis or order dissertation work, because this helps for receiving good level if you have dissertation.
Posted by: chopard watch Apr 18, 2010 @ 23:17
chopard watches and replica chopard tend to be coated in a simulation silver or tag heuer gold and replica rolex filled with breitling watches a much cheaper rolex watche smetal to save on cost.
Posted by: wholesale clothing Apr 19, 2010 @ 00:17
It is good!