To tired to focus on code at the moment, yet much to awake to sleep \o/. Been tinkering with an old project: tpsh. Mostly I've been polishing the codebase and doing a bit of refactoring; there's no a debug mode, which reduces some of the cruft that's crept in from testing. Sometime I also need to import the test scripts lol.

One of the changes, is adding support for running off the Strawberry Perl distribution rather than ActiveStates Perl distrio for Windows. The downside is, most of tpsh biggest problems have to do with Perls portability, namely Windows quirks. Gotta love'em.

Since the merge of the "Code generator", tpsh has supported a very limited subset of sh script. Not very usefully however, since the shell doesn't have a real concept of $? yet. Likewise there needs to be some changes in the handling of environment variables. Most ideal IMHO, is a tied hash wrapping $ENV (a magicly tied hash of environment variables) with the ability to hook reads and writes, etc. Preferably in a user-exposabe way that can be extended by shell functions, rather than being limited to the scripts own Perl code.

It's the more general case however, of having to build up the language rather than use its building blocks. The problemo isn't in maintainability, but in portability and development time.

Having reactored a chunk of the shells initialization code,  I know that I'll have to retackle the readline stuff.  To say that I hate Term::ReadLine would be an understatement. I've no love for the GNU Readline API either (I would use linenoise), but the GRL C API isn't as nasty in practice as the plugable Term::ReadLine system Perl uses. The only good thing I can say is that there is a semi-useless stub version included. Licensing issues of just having a GRL backend aside >_>. In my experience, mileage varies quite a lot between Perl readline pacakges, even within the scope of the fscking manual. It's just a load of crap.

Actually if there's a way of accessing the API  functions needed for unix termios / windows console, I would be tempted to throw out the fucking thing and just write a Term::ReadLine::SANE module based on linenoise (A C library). I'm more familiar with the unix parts of that.

Oooh, I've just had a little brain fart of interest.

For lack of anything more interlectually interesting to do with my sleep deprived mind, I've been thinking a bit about some coding I could do on tpsh; then this hit me. The shell expands ', ", and ` quotes using a simple table to map the symbols to appropriate transformations, why not use the same code for () and {} grouping?

Internally the look up table for quote expansions is hard coded into the principle tokenization subroutine, because that is the codes only designated purpose in the program; it didn't need to be more general, then being easily converted to something more generic. After getting a working implementation that I could drool over, I thought about posting a modified version on a forum, as a demonstration of how to do quote handling in config files and such.

Now I'm thinking of more places I can use that little blighter with a few minor changes lol.

Several hours of testing has shown where the problem lays in regards to the kinks in tpsh's basic control flow.... a strict ficken implementation lol.

Without a few extra ';' between language elements, the parser doesn't see the keywords as keywords, and instead they get fed into the code generator as arguments to the preceding keyword. Testing the same test code against the version of bash with MSYS1.0, their shell doesn't give a hoot, nor should any other bourne-style shell I have access to. So in a way, you could say my code took a stricter interpretation of the sh script syntax, then what is required (and desired).

Somehow this actually makes me feel better, lol.



Also something that needs working on, is dealing with crappy old-style paths like CP/M, DOS, and Windows NT use.

$ P:/Editors/Vim/vim-personal/gvim.exe
tpsh: command not found: P:/Editors/Vim/vim-personal/gvim.exe at S:\Visual Studio 2008\Projects\tpsh-dir\tpsh line 883.


$ cd P:

$ P:/Editors/Vim/vim-personal/gvim.exe
tpsh: command not found: P:/Editors/Vim/vim-personal/gvim.exe at S:\Visual Studio 2008\Projects\tpsh-dir\tpsh line 883.


$ /Editors/Vim/vim-personal/gvim.exe

$

It seems to have a bit of an issue about drive letters.


In more detail, the search_path() function that converts an external command name into the direct path to it (done in order to avoid a dependency on the systems shell, e.g. /bin/sh or %COMSPEC%), fails to find a valid file when the drive-letter: notation is used. Since the function returns undef to indicate program not found, simply put search_path() needs fixing. The change should be trivial though.

I managed to get some coding in today, maybe that's why I'm feeling a bit better. On the down side though, my cable connection seems to be running half as fast as normal lately, games were nearly unplayable for much of this afternoon for some reason.

I think my shells codegen branch has lived to the virtual extent of it's usefulness. Today I sorted a few minor things and enabled nested control flow for statements using the 'then' and 'do' keywords, e.g. control flow statements can contain control flow statements as well as commands within their block. There are still a few kinks to iron out but the most important are done; I'm currently uncertain if the remaining problem lays in the code generation phase or somewhere further up the processing chain, but I expect it lays in generating the perl code.

Basic conditional flow and looping is ready, which was the primary purpose of this branch of development: to explore implementing the shells scripting language by dynamically generating Perl code for it, as well as replacing the static command sequence executor by generating it as well.


Oddly, the main things that need doing in the short-term are actually script related, rather then interactive. But then again, tpsh has always been meant for interactive usage first and support for batch jobs as a secondary concern. Main points of lacking at the moment, is a concept of "exit status", in Bourne shell parlance the $? variable isn't used for a hill of beans yet. Likewise all $variable expansion is done on the fly during tokenization, so:

for X in 1 2 3; do echo $X; done

The value of $x in the loop body would be expanded before the code is evaluated, resulting in the following:

for X in 1 2 3; do echo; done

because X is not defined at the time the statements are being parsed, the expansion is "".



I haven't decided how this will be solved yet. Solving it however, is well outside the scope of the codegen branch. I would say the only thing left to do in the branch, is implement the break num and continue num to enable a way out of the while/until loops. The former will take a bit more testing but the latter, is not likely to be to hard. Perl supports a robust way of breaking out of loops that can be easily used. Once that stuff is done, I'll likely merge the branch into master and move on to other tasks.


Since I've been stuck working more often on the NT machine, development of tpsh gets a slightly higher priority lol.

In sleeping on it and having a job, that frankly leaves my mind free: I've come up with an idea for the tpsh problem.


The tokenizer is beautiful, the only problem is it does the expansions through a table of call backs. Now if I was to modify that so that it instead attaches meta-data to each token instead, those expansions could be delayed until later without major redesign. It will be necessary to redactor the lexer in order to handle the change in data structure. The code generator can then be tasked with completing any expansions during generation, by expanding lexemes to corrisponding code as it goes. After wards, re do the code generate into something more elegant and voila.... we have what it should've been in the first place!


A good nights sleep always helps the code[r]

A small shoot yourself in the foot, coders moment.

Being bored and lacking further ops I can get done before bed, I picked up on tpsh again. In look for a quick challenge, I noted that the git repo was still on the 'codegen' branch. Basically, a branch to test the idea of generating the execution code on the fly per command sequence.

As a quickie of interest, I picked up the generation phase for the for-loop. Then I hit a road block. Since my shell expands variables, globs, and aliases quickly during tokenization. The reason being, the input field separator ($IFS) and quoting rules determine how this shell splits text into 'words' or tokens for the execution mechanisms—you could say it's "On the way there", thus deals with it as it comes. Currently tpsh handles environment variables by fouling around with the programs own idea of the referenced environment variables (%ENV) without any distinction between exported and unexported variables. My intentions have been to use a more controllable interface for shell variables at a later date, since it is kind of a low-yield concern at this stage of development.


I see several choices:


a/ redesign how things work (obviously this is the whole story, lol), saving the issue until later when other components have matured to match.

b/ leave expanding environment variables (etc) until the last minute, I don't like this idea.

c/ have some way of retaining things that can not be confirmed until later, with an indicator to strip out or expand the remainders at the last minute (this gives me visions of ugly code)

d/ incorporate the code generator closer into the process, so that things get expanded only after they have been confirmed, but generated as soon as possible (a more multiple pass focused design comes to mind).

a is basically a worry-later, and see if other things that needs doing either fix or exacerbate the problem (this is a two edged flaming sword). b is possible but would take a lot of reworking crap, and IMHO result in an ugly LA phase and become prone to introducing bugs into the final results. c sounds simple enough at first glance but I do not see a method that I'm willing to live with. I worry about how easy d would could confuse readers, and what danger a slip up on it could do the results.


For now, I intend to not worry about the minor issue, until after variable handling matures. Because I really love how expand_quotes() works, and that is the best part of the whole program IMHO. Needless to say, tpsh has poor handling of shell/environment variables and has had it throughout its development, since growing the code can wait longer then the other parts.


Not to mention the fact that tpsh has mostly been developed under sleep deprivation in the first place.... lol

edit

In the time it took to submit the entry, type 'shutdown -p now', put away the computer, and take a quick leak: I came up with another solution. Give the code that expands variables an understanding of how they are defined, rather then only how they are referenced. Not only would checking if a referenced variable was just defined in the same set of input work with 'for X' and 'for X in ...' like constructs, it could also be used to implement the 'VAR=... ... command ...' syntax at a later stage :-)

The way expand_quotes() invokes the other expand_* procedures, we would need adjustment before the syntax of prefixing commands with variable settings could work, yet implementing the for-loop in it would be trivial, since anything that would cause the statement to get broken into an unusable token set before defining said variable and attempting variable expansion on it, would also be a usage that gets around for's keyword status!


Problem solved with less fuss, maybe? It is amazing what you can think of while taking a piss!

quick note to self

once tpshs implementation of shell script is more mature: transition the windows machine to running a shell script on init, rather then the \Startup\ system used by Windows NT, and compare performance.

more tpsh: control flow stuff

I've been trying to find a way of hooking in proper shell control flow keywords into tpsh, without uglifing the existing code. At the moment, tpsh understands executing singular command sequences, scripts, and a queue of command sequences. It's fairly easy to modify the parsing/lexing portions to adjust the internal data structures IAW control flow keywords, the problem is how to handle execution phase.

Originally, I had in mind setting up nested data structures and doing a delayed execution: evaluate the control flow statement and modify the data, then execute the remaining commands (e.g. if CMD0; then CMD1; else CMD2; fi, would execute CMD when the statement needs to be evaluated at execution phase, then reshape the strucure so that only CMD1 or CMD2 remains, and then feed that back into the executor).

Last night, I had an interesting idea... on the case specific code generation.

Shell control flow basically amounts to very simple if, while, for, and case statements; and the more modern until and select statements. Normal execution patterns amount to using a single command sequence (e.g. cat f0 f1 f2 f3 | sort > f), or a queue of such command sequences. Why not replace that executor with a section of code, that understands how to handle those as well as control flow (etc), and then generate the desired code to execute the result.


Exempli gratia:

tpsh_cgen( ( [ 'if', 'test command' ],
             [ 'then', 'other commands' ],
             ....
             [ 'else', 'other other commands' ],
             ....
             [ 'fi' ] )
)

might return something like:

sub { 
    if (evaluate the 'test command' and test the exit status)
    {
        execute the 'other commands'
    }
    else {
        execute the 'other other commands'
    }
}

and so on and so forth; so that if we call the generated code ref, we have a set of code that will execute the correct commands, whatever they may be, and with quite a lot less fuss.

merging the new code into master

After 3~4 days of coding, I've just merged the parserlexer branch back into master; I love coding :-)

 commit 9857e5e9556f31543075fb4a74350dbda97a42e5
 Merge: c9a8ae4... bb425eb...
 Author: Terry ....snip...
 Date:   Wed Apr 8 07:23:19 2009 +0000
 
     Merge branch 'parserlexer'
     
     The new parser, lexer, and quote expansion code (+ a few bugfixes) has
     finally been merged into the mainline of development (branch 'master').
     This marks the new sh_eval(), tpsh_parse(), expand_quotes, tpsh_lex()
     functions in such a way that they should be considered 'stable' for
     general usage.
 

some nice things that come with this:

a sane way of quoting stuff; but not sh compatible ('\'' = ', not an error), more then one set of quotes on the line; and things like `cd /foo; vi bar` finally work xD. In the course of the necessary bug smashin' for the merge, I've also cleaned out a few pains in the todo file, that have been there since last month++ lol.


things that remain to be done: pluggable completion; make completion play nicer with perl/gnu readline backends; restore support for pipes (critical); handling of keybinding (likely painful across perl/gnu/zoid Term::ReadLine backends; but at least zoid is nice...). In the more long term: control flow, (subshells), better `handling`, {anonymous macro/grouping}, more advanced I/O redirections (i.e. only >, >>, and < currently work lol); making `fc -l` and `history list` use a format for display rather then print(); make &do_getopt able to be configured by callers (so fc can accept negative indexes, etc); give a way to turn off shell options (set - and set + currently turn on, only lol); and who knows what else that I can't remember atm.


and to abuse {erls idea of objects and verbs: eval { $spidey sleep $now };

tpsh: test of expand_quotes()

$ echo 'hi bye' foo "$USER" and "~" or ~
expand_quotes ': echo  | hi bye |  foo "$USER" and "~" or ~
expand_quotes ":  foo  | $USER |  and "~" or ~
expand_quotes ":  and  | ~ |  or ~
hi  bye foo $USER and ~ or /usr/home/Terry
$

# note: 
#       the 2 spaces /displayed/ between hi and bye are a bug in
#       tpsh; echo'ing things to file via I/O redirection works
#       properly. "$USER" is not expanded because expand_parameters()
#       still needs adjustments.
# 

tpsh_parse invokes expand_quotes() to break up its input line based on the shells quoting rules; and proceeds to go about it's business. tpsh_lex() then accepts the token buffer and begins building a new data structure from it. The tokens from tpsh_parse get analyzed and reassembled "on the quotes", i.e. it will do it's check on 'hi\ ' and 'bye' and the rest as separate elements; then reassemble the argument vectors as an array reference: becoming 'hi bye' again. (id est quote expansions add escapes to tell the lex phase where to rejoin things) After everything is said and done between parse and lex, the queue like data structure is ready, the argument vectors contained there in are ready to be mapped onto resolve_cmd() calls for execution.


To hunt down any other booboos in the expand_quotes() subroutine, I've made it display it's work, so I can see how it detects what when testing the shell. basically as "expand_quotes QUOTE: unquoted | quoted | remainder".


As one can guess from what the above shell snippet implies: quoting is handled recursively. Because I'm used to languages with finite stack space and no reliable tail call optimizations; I almost never write recursive functions of any kind, whether they are tco'able or not. Algorithmically, expand_quotes() is a very simple procedure.


It expects to be called with an input line; and treats multiple arguments accordingly (for now). Internally a dispatch table and token stack are maintained; the table contains references to anonymous subroutines, to which the scanned elements are delegated to for the proper expansions.

If no quotes are detected on the line, return the result of expanding it with the default delegate (for unquoted text).

Otherwise break the line on the first set of (matching) quotes.

Any text defined before the beginning quote must be unquoted; apply the default expansion from from the table.

The text between the matching quotes is quoted, apply the appropriate expansion form the table (i.e. ', ", or `).

Any text remaining after the matching quotes may or may not be quoted; invoke expand_quotes() on the remainder to find out, and apply the result.

Each expansion applied is pushed onto the token stack in the escaped form it expanded to (i.e. "'hi bye'" becomes "hi\ bye"), and the stack is returned to the caller once processing is completed.


With refactoring, the procedure could likely be made tail recursive but I don't think perl does TCO. Either way, the users fingers or (likely) the machine generating the inputs should run out of stack space before tpsh could pop a cork at the number of quotes lol. An earlier design for expand_quotes() had more in common with finite state machines (in so far as I've seen them implemented), but was a lot more contorted then expand_quotes()' present shape :-/.

Current bugs are handling nested escaped quotes or multiple empty quotes (the spliter) and removing unquoted quotes (addition to delegate sub for unquoted text).

# bugs in expand_quotes
$ echo 'foo \"bar'
expand_quotes ': echo  | foo \"bar |
foo  "bar
$ echo "foo \"bar"
expand_quotes ": echo  | foo \ | bar"
foo   bar"
#
# correct result would have been equal to the previous command
#
$ echo '' "" '' "" '""' '' "" '"' "'"
expand_quotes ': echo  |  |  "" '' "" '""' '' "" '"' "'"
expand_quotes ":  " |  ''  | " '""' '' "" '"' "'"
expand_quotes ': "  | "" |  '' "" '"' "'"
expand_quotes ':  ' |  ""  | "' "'"
expand_quotes ": "'  | ' |
"  ''  " "" '  ""  "' '
#
# correct result would have been:     ""   " '
# at least, that's how all bourne based shells I 
# know about treat it; I would prefer: "" " '
# i.e. without leading whitespace.
#

For some reason this makes me curious, has anyone ever explained why shell syntax allows "\"" but not '\'' ? (the results being " and unclosed quote /or syntax error respectively)


When trying to solve a programming problem, generally I try the most simple solution before I try something more complex: and then evaluate a neater method. I consider the implications solutions have on efficiency, but that is trying to avoid shooting myself in the foot later, rather then trying to optimize the code for a machine.


Some how, I think expanding quotes is just naturally recursive in my crazy brain :-D.

EDIT

commit aeac14bd177a93b84c138a0c62e2cda49e5fe15c
Author: Terry <***snip***> 
Date:   Tue Apr 7 22:24:35 2009 +0000

     bugfix: parameters now expand within quotes via expand_quotes and may be escaped

commit 089fda7cca0049dcabdc8b9659f94dcae417074b
Author: Terry <***snip***> 

     bugfix: escaped quotes witihn quotes and multiple quotes handled correctly

     previous behaviour:

     $ echo 'foo \"bar'
     expand_quotes ': echo  | foo \"bar |
     foo  "bar
     $ echo "foo \"bar"
     expand_quotes ": echo  | foo \ | bar"
     foo   bar"
     $ echo '' "" '' "" '""' '' "" '"' "'"
     expand_quotes ': echo  |  |  "" '' "" '""' '' "" '"' "'"
     expand_quotes ":  " |  ''  | " '""' '' "" '"' "'"
     expand_quotes ': "  | "" |  '' "" '"' "'"
     expand_quotes ':  ' |  ""  | "' "'"
     expand_quotes ": "'  | ' |
     "  ''  " "" '  ""  "' '
     $

     new behaviour:

     $ echo 'foo \"bar'
     expand_quotes ': echo  | foo \"bar |
     foo  "bar
     $ echo "foo \"bar"
     expand_quotes ": echo  | foo \"bar |
     foo  "bar
     $ echo '' "" '' "" '""' '' "" '"' "'"
     expand_quotes ': echo  |  |  "" '' "" '""' '' "" '"' "'"
     expand_quotes ":   |  |  '' "" '""' '' "" '"' "'"
     expand_quotes ':   |  |  "" '""' '' "" '"' "'"
     expand_quotes ":   |  |  '""' '' "" '"' "'"
     expand_quotes ':   | "" |  '' "" '"' "'"
     expand_quotes ':   |  |  "" '"' "'"
     expand_quotes ":   |  |  '"' "'"
     expand_quotes ':   | " |  "'"
     expand_quotes ":   | ' |
     "" " '
     $

recent shell efforts

well, the foundation has been laid down for the next phase of tpsh development. Branch 'parserlexer' is basically setup to deal with the changes in tpsh_parse, tpsh_lex, and the switch from singular command line execution to enqueued command execution.

The new quote handling is actually quite a bit better, if buggy for now. The sh_eval() functions mostly become dead weight; coupled with the behaviour changes in tpsh_{parse,lex} behaviour the source, ., and eval built-ins (and anything relaying on them) are temporarily broken; as is tpsh -c 'cmdstr' until things are further integrated. Pipes also no longer work, since the command resolution doesn't know how to deal with the command queue yet lol. Fixing 1 subroutine should will fix most breakages.

The idea is more or less that a command like

$EDITOR -o f1 f2 f3; cat f1 f2 f3 | sort -args | sed 's/x/y/g' > /tmp/q

becomes this:

     ( ['vim', '-o', 'f1', 'f2', 'f3'],
       ['cat', 'f1', 'f2', 'f3, '|'], 
       ['sort', ',-args', '|'], 
       ['sed', 's/x/y/g', '>', '/tmp/q'] )

and the trailing '|' symbols would be used to indicate that the current element should be joined with the next (in a non technical sense that is) until the end of pipes is reached; recreating the pipeline (in so far as what happens).


The line is parsed into tokens, then analyzed and formed into a more interesting set of elements like the above array of array references; where the array refs are the argument vector (argv) of the commands to be passed onto pexec() or other suitable function. Previously the line was just parsed and dropped onto resolve_cmd() to figure out if it's a pipe based, i/o redirection based, built-in, or external command; based on the scalar line or argument vector resulting from expansions.


the master branch remains the stable line for now until this topic branch is finished with.

It's been a pretty good day so far. Got stuck getting up early for a shopping trip, but hey at least I got some doughnuts out of the deal lol. Oy, I'm going to end up doing press ups more often 8=). Two bags of powdered doughntus my absolute favorite xD.

Ducked into Proving Grounds #1, and joined Spawn, Ez, Hostile, and a few pubs for some games. Man, it's been insane today in the servers. Those I don't give a crap if I hit anything, it's time to empty the magazine kind of moments - like a bad zombie flick, swarms of tangos out for blood. After a bit of a break to work on tpsh, I ended up in Proving Grounds #3 with Duke and a couple others joined: but still groups 3-5 tangos haunting the halls. Well actually that's not to bad, in RvS it was more like 4-7 tangos at a time... hehe. One odd thing, this time out in SWAT 4, I got stuck in the heavy plates. Normally I hate body armour that restricts movement, especially in games like SWAT where heavy armour slows you down, and doesn't stop a patato gun, let along bullets. I think the suspects must've gotten scared -- last man standing, and feeling like a land battleship, but surviving without injury ;-).


tpsh gained the sh derived `tpsh -c "commands"` behaviour today. Command completion and history features have evolved quite nicely. Really what needs working on is the shells lexical analysis. I figure for setting it up as my login shell, I'll compile a small C program that sets up PERL_RL to load a suitable Term::ReadLine backend before exec'ing tpsh.


All in all, not a bad day; but not very furfilling either :-/

Code monkey go to bed...

It's been a rather slow day, but somewhat productive (only about 10 or 11 commits :@). My families made sure that I've had a throbbing headache most of the day... rat fuckers! But at least there is some work to show for it. tpsh now has a concept of $PATHEXT based on cmd.exe's %PathExt% variable.

Windows is ruled by file extensions, while UNIX could care less about them; so really one of the few good things about Microsoft's cmd.exe is you can tell it what file extensions should be "understood", i.e. so you can type 'notepad' instead of 'notepad.exe'. Since tpsh is modeled after the standard Unix sh, it's mostly oblivious to file extensions: it cares about names. However it is virtually _impossible_ to use Windows from the CLI level without implementing something like PATHEXT or typing yourself into a nightmare (winxp/cmd.exe is actually good at making you do that, compared to a unix/sh).

For better compatibility with a Windows environment, tpsh now implements it's (my) own concept of the feature, complete with a 'pathext' option (default on) to toggle the functionality. The main reason tpsh does this, is so I can type 'gvim' when I mean 'C:\Path\to\gvim.bat', the fact that I'm used to typing 'vim' is aside the point lol. (I rarely use gvim off win32 b/c of diffs between nt and unix cli)


About the only time I use file extensions is when forced (Win32), when ideal (.zip, .tar, etc), when saving text with CRLF for new line indicators (notepad friendly .txt), multimedia files (.png, .ogg, etc), or when deploying crap to a Windows machine lol. So it's not an important thing for me; just a time saver.




Another thing I sorted out is what I call the "hash separator" for environment variables. UNIX shells by convention separate values like $PATH with a ':', e.g. '/bin:/usr/bin', but DOS and related bastards use ';' for things like %PATH%, e.g. 'C:\Windows;C:\Windows\System32'. Because many older operating systems use a 'letter:path' style for paths anyway, there is no universally portable default setting. Since virtually all operations involving a variables like PATH, CDPATH, and ENV (an extension used in tpsh, which I've never seen another shell use) involve a hash table, I call the mark the 'hash-sep' for short. One of today's changes was exposing the hash-sep directly to the user.


The default hash-sep is ';' under MSWin32, DOS, and OS/2 and ':' otherwise, I've no clue what the hell VMS uses and don't have access to it, so no worries yet ;-). Whenever the user changes the HASHSEP environment variable, the next time an operation that relies on it (basically hsplit(), short for hash-sep split()) the shell compiles it down to a suitable regular expression to save time on future commands.

setenv HASHSEP '/'
rehash

the rehash will cause '/' to compile via qr for to speed up later splits; and due to the change of hashsep, causes the internal cache of $PATH to switch from the default ':' or ';', to '/', which is probably not something anyone wants to do, but being able to fiddle with it can be useful for scripting reasons ^_^. Not to mention the fact that it makes concatenating things like PATH/CDPATH settings more portable when the environment requires something different.



Another bit of today's work, was setting it up so that the history built-in now displays the correct line numbers. I want to setup builtins to save/load data from HISTFILE; so that command history is not lost between sessions - I've always had a bit of an itch about how most shells deal with it; wonder what tpsh might do hehe. Not sure about the $LINENO yet, I never really use it in scripting or interactive usage that much: will probably take the single unix specification into consideration on $LINENO.



If I didn't have to so much crap for tomorrow, I could probably have half the manual done tonight and still get to bed before 0600 local. But no.... work early, work long, to be driven nuts after work, and probably end up PTFO instead of coding the night way.


*sigh*.



my idea of a great vacation: a Ferrari, laptops, solar panels to power them, hot date, and a tropical island with a sunny beach ;-)

An epiphany!

3. a sudden, intuitive perception of or insight into the reality or essential meaning of something, usually initiated by some simple, homely, or commonplace occurrence or experience.

I actually thanked my mother for dragging my out on a shopping expedition today, because I figured out sometime a ways "down" my todo list: implementing control flow in tpsh. I guess you can take the programmer away from the code, but you can't take the code away from programmer ^_^ ^_^ ^_^.


Earlier I recorded that the goal was for tpsh to use a queue of commands to execute rather then going by lines, the idea being:

`cmd1; cmd2; cmd3`

would parse into a list like '(cmd1, cmd2, cmd3') which tpsh would then walk, eval, and execute the result in sequence (i.e. first in, first out). Then it occurred to me, if the shell will go that route for lexical reasons: why not implement the shells control flow operators as keywords that manipulate that queue directly? So that conceptually, a shell snippet like:

cmd0
if [ expr ]; then
  cmd1
  cmd2
else
  cmd3
  cmd4
fi
cmd5

would be inserted into the queue in a suitable manor and passed to the call back, so if the queue looked something like this:

( cmd0, if [ expr ], cmd1, cmd2, else, cmd3, cmd4, fi, cmd5 )

the call back would receive that relevant portion of the queue, (i.e. queue₁ through queue₇), evaluate the test and return the appropriate portion to be spliced into the queue, in this case:


( cmd0, cmd1, cmd2, cmd5 ) or ( cmd0, cmd3, cmd4, cmd5 )

depending on whether `[ expr ]` evaluated as true or false.



Damn, this so makes me want to straighten up the necessary parts of tpsh... now if only I didn't have to work tomorrow from morning to whenever I drop over undead, or pass out lol. These kinds of things have always fascinated me about programming, but life has never offered much chance to study compilers, virtual machines, and interpretors... let along create something like tpsh, that requires implementing a scripting language :\. The one good thing, despite the ups/downs and expressiveness of sh script, it's really an easy language for a human to understand.

hmm, there was something I wanted to add a `set -o option` for, but I can't remember what the heck it was lol.

man it's been a slow day... I've only made about 15~16 commits on tpsh, one of which was merging branches lol. The main focuses of today's tinkering has been minor bugfixes (mostly related to and tweaking the completion function.

Previously my completer would just take a quick approach, but always doing completions for built in commands, stored macros, known programs, and filenames. Now it has some notion of *what* it should complete.

$ b^I -> assume it can be anything
$ echo b^ -> assume it's a file (formally a full eval() was done)
$ builtin l^I -> complete built-in commands
$ alias x -> complete to names in %Macros.



When I get the time, I want to make the completion function have a more hot-plug-in nature to it. A pre-completion hook, completion hook, and post-completion hook; the pre getting to modify the thing before completion, the completion hook being able to totally replace do_completion() aside from the pre/post hooks; and the post-hook having a change to modify the completions being returned. The current working idea is: a TPSH_WHAT_HOOK environment variable being set to A. perl code to generate a CODE ref from, or B. an external program to delegate to. (e.g. TPSH_COMPLETION_HOOK="compl.rb" would pass compl.rb the necessary data, and expect the program to generate a \n delimited list on its stdout for tpsh to parse). Odds are there will be more variables to tune it with later, as well as the API avail. being documented in the manual page My main reason for desiring this style of dynamically configurable completion is so I could do things like, seting up my tpshrc to "ignore" any 'lost+found' entries in filename completions; i.e. /usr/local/l^I expands to /usr/local/local not /usr/local/lo - completion list for local / lost+found. I know the zsh (which I use a lot) is noted for it's configurable completion and spelling correction, but honestly I've never customized it beyond the lines zsh's setup program added to my rc, let along RTFM'd about it lol.

Making it easy to add command sensitive completions through the hooks would be nice too, e.g. `man 2 foo^I` and have a hook spot the man 2 and go about completing foo to manual pages in section 2, hehe. (the only zsh feature I abuse, actually lol)


Really what needs doing atm in tpsh, is cleaning up the lex/eval subroutines and merging them and a few related changes with resolve_cmd(). Basically the principal adjustment needed is to go from line-based handling of commands, to (properly) breaking them lexically for storage into an execution queue. Generally though I like how stuffs coming along, except for a few odds and ends here and there.



At one point,tpsh relied on a list of "this command marks something to skip macro expansion on", a quick solution until expand_aliases() could be rewritten. One night I found myself working on a scratch file with commands to modify said list (add, rm, and show the non expanding commands list via expr or array slice/index). That's when I threw my hands up in disguest, and decided tomorrows job would be fixing expand_aliases (note to self: s/aliases/macro/). Imagine if you tried to type 'unalias ls' and the shell expanded ls to it's aliased value, that kinda stuff. Kind a cool to be able to configure that, but kinda wrong to need a linear look up on each pass through the loop (ugh!).


To add built-in commands to manipulate said list rather then fix the problem, just was not something I was NOT willing to do... because to my eyes, it would be like a pox on my honour as a programmer: to gamma-weld such a cheap trick into place with built in commands to configure and abuse it rather then fix it -- thus the macro expansion system got re-implemented, much more correctly this time. (didn't have time to sort positional params and what nots).




I am also VERY glad that tpsh is being developed with git, rather then cvs! (I've also spent part of tonight learning more about gits plumbing)

Been working over SSH from SAL1600 today, since my rooms flooded out with cloths racks :\. This crappy wireless ain't helping either, just to keep things stable I've had to drop from AES encryption to Blowfish and add compression... either the signal utterly blows these days, or PuTTY must leak a lot of resources me thinks.


I cloned master in /srv/git/Projects/tpsh to /tmp/tpsh to do a little work, when surprise surprise... OpenBSDs perl barfed at Getopt::Long::GetOptionsFromArray. A cursory inspection of installed Perl modules around the network & change logs, showed my worst fear was right: it's an experimental function added 2 years ago. After inserting a banda-id, and doing quite a bit of testing to avoid regressions... I also fixed a few other bruises and found a few things Perl 5.8.8 / use warnings didn't like very much.

The birth of the "porting" branch, lol.


After taking care of pushing that out to the shared-repos, I checked out master and implemented the eval built in, so before I start coding tonight on my laptop I'll need to update the master branch there from the one on the server, and merge in the changes from the porting branch.


Currently I've got three versions of Perl installed, 5.8.9 on the FreeBSD laptop (the authoritative git repository), 5.10.0 (activeperl) / 5.8.8 (from msys-git) on Windows XP, and 5.8.8 on the OpenBSD machine (home of the shared repository / insurance policy).



if tihs infernal wireless goes out once more.... I might just start working from my laptop on the couch, and the hell with this P.O.S. At least then, Iw ouldn't have to SSH to get work done ^_^.

commit b841dc4954c24d0abea43daf407b6bf70e1c450b
Author: Terry ******* ****** <***********@****.***>
Date: Wed Mar 25 07:15:31 2009 +0000


massively improved alias expansions

aliases now expand recursively until resolved or aborted. A circular
alias like x = y; y = z; z = x; will resolve x to x when it hits z. The
expansion of aliases should behave more or less as desired, but without
positonal paramter support

tpsh:
$ alias x='y -opts P'
$ alias y='z -flags PP'
$ alias z='echo z PPP'
$ x one two three
z PPP -flags PP -opts P one two three
zsh:
$ alias x='y -opts P'
$ alias y='z -flags PP'
$ alias z='echo z PPP'
$ x one two three four
z PPP -flags PP -opts P one two three four

At the moment the shell local()'s %ENV before each expansion, and will
likely set 0..$#, $#, $@, and $* accordingly someday; but currently does
not use %ENV for anything. In order to allow macros to change the
environment, we can't just local() %ENV to implement positional params
for macros, but it will be a suitable stop-gap until done fully.

note: alias, macro, and function all revolve around %Macros and
&expand_aliases for macro expansion.

the next big chore is improving the code that invokes expand_aliases(), lol.

another (big) section of the manual written, ENV processing, and an initial implementation for the history built-in among a few other things is done.


One interesting thing, usually sh only allows a single file in $ENV, and some versions of sh don't even understand it period! In tpsh, as an interesting extension $ENV is treated like $PATH, in so far as ENV=/etc/tpshrc:/usr/local/etc/tpshrc:~/.tpshrc would cause tpsh to source an rc file in /etc, /usr/local/etc, and then the users home directory.

(Because of old OSes using drive letter:path like C:\Windows, under such OS tpsh uses ';' instead of ':' to separate things like $PATH)

How the code comes out, when you're ready to pass out

# message for commit 041a7343eb452b827dbd97a0c82c8538597f86f6:
#
#   read built-in command implemented


sub read_bin {

    my ($prmpt, $time);
    my $line = "";
    my @argv = @_;
    my %opts = ( 'p=s' => \$prmpt,
                 't=s' => \$time,
                 'e'   => sub { "no-op" },
               );

    do_getopt(\@argv, %opts);
    unless (@argv) {
        warn "I can't read into thin air!";
        return 0;
    }

    if ($prmpt and -t *STDIN) {
        print $prmpt;
    }

    eval {
        # remove custom die for ease of error check/report
        local %SIG;
        $SIG{__DIE__} = sub { die @_ };
        $SIG{ALRM}    = sub { die "timed-out\n" };
        if ($time) {
            # an s, m, or h suffix causes sleep for sec, min, or hour
            #
            if ($time =~ /^(\d*)([smh])/) {
                if ($2 eq 's') {
                    $time = $1;
                } elsif ($2 eq 'm') {
                    $time = $1 * 60;
                } elsif ($2 eq 'h') {
                    $time = $1 * 3600;
                } else {
                    warn "internal error on ", __LINE__;
                    # NOTREACHED
                }
            }
            alarm $time;
        }
        chomp($line = );
        alarm 0 if $time;
    };
    if ($@) {
        warn $@ unless $@ eq "timed-out\n";
        # on time out, init the vars to empty strings
        @ENV{@argv} = ('') x scalar @argv;
        return 0;
    } else {
        # set each var to the words
        #
        # XXX because ifsplit has no notion of a &split 'LIMIT'
        #     if we used ifsplit here instead of a manual split,
        #       read x y
        #       foo bar ham
        #     would set $y to 'bar' instead of 'bar ham'
        #
        my $ifs = defined $ENV{IFS} ? qr/[$ENV{IFS}]/ : qr/\s/;
        @ENV{@argv} = grep !/^$/, split $ifs, $line, scalar(@argv);
        return 1;
    }
}

from the manual page updated in commit b1317d7e6e7f91b6c3a2650f44cd4f425e381d42 with message:

read built-in command documented

and blockquoted here using the pod2html output

read [-p prompt] [-t timeout] variable ...]

Read a line from standard input, split by fields, and assign each field to the indicated variables. If the number of variables is less then the number of fields, the remaining fields will be stored 'as is' in the last variable. If there are more variables then fields, the excess variables will be undefined. A prompt may be printed before reading input, by using the -p option. The -t option may be used to specify a timeout in which to abort the operation, should the user take their sweet time about pressing CR. The timeout value can take an optional s, m, or h suffix to denote seconds, minutes, or hours. If no suffix is given, s will be assumed.

It's not the greatest... but hey, I ain't had any sleep since this mornings roll out... lol. It'll do fine for an initial implementation, until I've actually got a functioning brain to deal with it :-P