: ) wonderful world ( :

the metasyntactic variable

join & filter multiline data records with sed

leave a comment »

Grouping with sed:

$ cat nrs.txt
01
02
03
04
05
06
07
08
09
10
11
12
$ cat nrs.txt | sed -n -e 'N;s/\n/ /g;p'
01 02
03 04
05 06
07 08
09 10
11 12
$ cat nrs.txt | sed -n -e 'N;N;s/\n/ /g;p'
01 02 03
04 05 06
07 08 09
10 11 12
$ cat nrs.txt | sed -n -e 'N;N;N;s/\n/ /g;p'
01 02 03 04
05 06 07 08
09 10 11 12
$ cat nrs.txt | sed -n -e 'N;N;N;N;N;s/\n/ /g;p'
01 02 03 04 05 06
07 08 09 10 11 12
$

 

Good for the following task:

$ cat entries.txt
entry-1-data-1
entry-1-data-2
entry-2-data-1
entry-2-data-2
$ cat entries.txt | sed -n -e 'N;s/\n/ /;p'
entry-1-data-1 entry-1-data-2
entry-2-data-1 entry-2-data-2
$

 

Two long groups, removing elements (1st, 2nd):

$ cat nrs.txt | sed -n -e 'p;n'
01
03
05
07
09
11
$ cat nrs.txt | sed -n -e 'n;p'
02
04
06
08
10
12
$

 

In general case you have groups of length _k_ and the starting pattern is ‘n;n;…;n’. There’s k-1 number of letter n here. You can place letter p around the n-s, which is exactly k possibility. If there’s a p in position k0 then the k0-th element will be printed out.
So if you have 6 long groups and you want every fifth element:

$ cat nrs.txt | sed -n -e 'n;n;n;n;p;n'
05
11
$

Written by grault

June 25, 2009 - 3:54 pm at June 25, 2009 - 3:54 pm

Posted in command line, linux, script

office open xml file sample

leave a comment »

Here’s the spec.

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<?mso-application progid="Word.Document"?>
<pkg:package xmlns:pkg="http://schemas.microsoft.com/office/2006/xmlPackage">
  <pkg:part
    pkg:name="/_rels/.rels"
    pkg:contentType="application/vnd.openxmlformats-package.relationships+xml">
    <pkg:xmlData>
      <Relationships
        xmlns="http://schemas.openxmlformats.org/package/2006/relationships">
        <Relationship
          Id="foo"
          Type=
"http://schemas.openxmlformats.org/officeDocument/2006/relationships/officeDocument"
          Target="/word/document.xml"/>
      </Relationships>
    </pkg:xmlData>
  </pkg:part>
  <pkg:part
    pkg:name="/word/document.xml"
    pkg:contentType=
"application/vnd.openxmlformats-officedocument.wordprocessingml.document.main+xml">
    <pkg:xmlData>
      <w:document
        xmlns:w="http://schemas.openxmlformats.org/wordprocessingml/2006/main">
        <w:body>
          <w:p>
            <w:r>
              <w:t>Hello World</w:t>
              <w:br />
              <w:yearLong />
            </w:r>
          </w:p>
        </w:body>
      </w:document>
    </pkg:xmlData>
  </pkg:part>
</pkg:package>

Written by grault

June 25, 2009 - 12:43 pm at June 25, 2009 - 12:43 pm

Posted in windowns

cumulating minutes begun

leave a comment »

$ cat seconds.txt
120
123
$ cat seconds.txt | sed 's/$/ 60 ~ 0 !=r +/' | sed '1i[1+] sr 0' | sed '$ap' | dc
5
$

Written by grault

June 24, 2009 - 4:11 pm at June 24, 2009 - 4:11 pm

Posted in command line, linux, script

compiler with flex & bison in 77 lines

leave a comment »

Please note that you cannot copy-paste the Makefile below.

afroid-laptop% ls
lexer.l  Makefile  parser.y
afroid-laptop% cat * | wc -l
77
afroid-laptop% cat -A Makefile
all:^Icompiler$
$
compiler:^Iparser.tab.o lex.yy.o$
^Igcc -o $@ parser.tab.o lex.yy.o -lm -lfl$
$
parser.tab.o:^Iparser.y$
^Ibison -dv parser.y$
^Igcc -c parser.tab.c$
$
lex.yy.o:^Ilexer.l parser.tab.o$
^Iflex lexer.l$
^Igcc -c lex.yy.c$
$
clean:$
^Irm -rf lex.yy.c lex.yy.o parser.tab.o parser.tab.c parser.tab.h parser.output compiler$
afroid-laptop% cat lexer.l
%{
#include
#include "parser.tab.h"
%}

DIGIT [0-9]
ALPHA [a-zA-Z]
WS [ \n\t]

%%

{DIGIT}+ { yylval.intval = atoi(yytext); return NUMBER; }
{ALPHA}+ { yylval.id = (char *) strdup(yytext); return IDENTIFIER; }
{WS}+    {}

%%
int yywrap (void)
{
}

afroid-laptop% cat parser.y
%{
#include
#include

extern FILE *yyin;
extern FILE *yyout;
%}

%union
{
  int intval;
  char* id;
}

%start S
%token  NUMBER
%token  IDENTIFIER

%%

S:
  /* empty */
| IDENTIFIER S NUMBER { fprintf (yyout, "let %s = %d;\n", $1, $3); }

%%

main (int argc, char **argv)
{
  char out[1024];
  ++argv; --argc;
  yyin = fopen (argv[0], "r");
  strcpy(out,argv[0]);
  strcat(out,".out");
  yyout = fopen (out, "w");
  yyparse ();
  printf ("Parse Completed\n");
}

yyerror (char *s)
{
  printf("error\n");
}
afroid-laptop% cat > test.src << EOF
heredoc> a b c 1 2 3
heredoc> EOF
afroid-laptop% make
bison -dv parser.y
gcc -c parser.tab.c
flex lexer.l
gcc -c lex.yy.c
gcc -o compiler parser.tab.o lex.yy.o -lm -lfl
afroid-laptop% ./compiler test.src
Parse Completed
afroid-laptop% cat test.src.out
let c = 1;
let b = 2;
let a = 3;
afroid-laptop% 

Written by grault

June 4, 2009 - 10:25 pm at June 4, 2009 - 10:25 pm

Posted in notes, parser

core context oriented programming with ContextL

leave a comment »

ContextL is a part of the Closer project with contributions from Pascal Costanza and probably others. As per the code repository, it’s actively developed.

(defpackage :a-test
  (:use :cl))

(in-package :a-test)

(contextl:define-layered-class
    bar ()
    ((value :initarg :value :layered-accessor bar-value)))

(contextl:define-layered-function bar-incf (f))

(contextl:define-layered-method
    bar-incf ((b bar))
    (incf (bar-value b)))

(contextl:deflayer log-layer)

(contextl:define-layered-class
    bar :in log-layer ()
    ())

(contextl:define-layered-method
    bar-incf :in log-layer :around ((b bar))
    (progn
      (princ (format nil "~A is incfed" b))
      (call-next-method)))

(defparameter +a-bar+
  (make-instance 'bar :value 11))

 

And now, the REPL:

A-TEST> +a-bar+
#<BAR {BEB2581}>
A-TEST> (bar-value +a-bar+)
11
A-TEST> (bar-incf +a-bar+)
12
A-TEST> (bar-value +a-bar+)
12
A-TEST> (contextl:with-active-layers (log-layer) (bar-incf +a-bar+))
#<BAR {BEB2581}> is incfed
13
A-TEST> (bar-value +a-bar+)
13
A-TEST>

Written by grault

May 29, 2009 - 11:23 pm at May 29, 2009 - 11:23 pm

Posted in lisp

ms access relations into graphviz

leave a comment »

Today is a scripting day : )

If you have to maintain a large MS Access database (manipulate data in it) with complex relationship matrix, the built-in relations diagram is quite useless. This is because Access won’t compute a proper layout. It just displays the nodes (the tables) next to each other and draw all the edges. Period. An idea is to dump these relations into a graphviz .dot file and let graphviz do the layout. This solution is very useful if you want to understand a complex database represented in Access.

C:\Users\grault\Desktop>dir/b test.accdb
test.accdb

C:\Users\grault\Desktop>type dumpRelationship.vbs
Set objFSO = CreateObject("Scripting.FileSystemObject")
Set objAcc = CreateObject("Access.Application")
fileName = objFSO.GetAbsolutePathName(WScript.Arguments.Item(0))
outName = objFSO.GetAbsolutePathName(WScript.Arguments.Item(1))

objAcc.OpenCurrentDatabase(fileName)
Set outFile = objFSO.CreateTextFile(outName)

outFile.WriteLine ("digraph G { graph [rankdir=""LR""];")
For Each tbl In objAcc.CurrentDb.Tabledefs
   If LCase(Left(tbl.Name, 4)) <> "msys" Then
     entry = """" & tbl.Name & """ [label=""<" & tbl.Name & "> " & tbl.Name
     For Each fi In tbl.Fields
       entry = entry & "| <" & fi.Name & "> " & fi.Name
     Next
     entry = entry & """ shape=""record""];"
     outFile.WriteLine(entry)
   End If
Next

For Each rel In objAcc.CurrentDb.Relations
  For Each fld In rel.Fields
    arrow = """" & rel.Table & """:" & fld.Name & " -> " & _
            """" & rel.ForeignTable & """:" & fld.ForeignName & _
            " [arrowhead=""odotodot"" arrowtail=""orboxrtee""];"
    outFile.WriteLine(arrow)
  Next
Next

outFile.WriteLine("}")
outFile.Close
objAcc.CloseCurrentDataBase
objAcc.Quit

C:\Users\grault\Desktop>dumpRelationship.vbs test.accdb test.dot

C:\Users\grault\Desktop>type test.dot
digraph G { graph [rankdir="LR"];
"Table1" [label="<Table1> Table1| <ID> ID| <Desc> Desc" shape="record"];
"Table2" [label="<Table2> Table2| <ID> ID| <Ref1> Ref1| <OtherDesc> OtherDesc"
shape="record"];
"Table1":ID -> "Table2":Ref1 [arrowhead="odotodot" arrowtail="orboxrtee"];
}

C:\Users\grault\Desktop>dot -Tpng -o test.png test.dot

(dot.exe:6480): Pango-WARNING **: couldn't load font
"Times Roman Not-Rotated 14 ",
falling back to "Sans Not-Rotated 14", expect ugly output.

C:\Users\grault\Desktop>

 

the result

the result

Written by grault

May 15, 2009 - 4:14 pm at May 15, 2009 - 4:14 pm

Posted in notes, script, windowns

concatenate or merge docx Word files dot vbs

leave a comment »

In case you have generated Word files which are parts of a future “master” document and also you want to automate the process of assembling them together by writing some sort of build script, the following snippet is probably useful. Generating documents is good to avoid redundancy. In case you have to have a document in which parts can be computed based on other parts (with also the miserable contraint on document format and production environment) , it’s good to consider splitting the document, generating the parts and assembling them at the end. And now, the code:

C:\Users\grault\Desktop\a>dir/b
a1.docx
a2.docx
build.vbs

C:\Users\grault\Desktop\a>type build.vbs
Set objWord = CreateObject("Word.Application")
Set objFSO = CreateObject("Scripting.FileSystemObject")

Set objDoc = objWord.Documents.Add()

objDoc.ActiveWindow.View.Type = 2 'wdOutlineView
For I = 1 To WScript.Arguments.Count - 1
   objDoc.Subdocuments.AddFromFile _
     (objFSO.GetAbsolutePathName(WScript.Arguments.Item(I)))
Next

objDoc.SaveAs(objFSO.GetAbsolutePathName(WScript.Arguments.Item(0)))
objWord.Quit
C:\Users\grault\Desktop\a>build.vbs out.docx a1.docx a2.docx

C:\Users\grault\Desktop\a>dir/b
a1.docx
a2.docx
build.vbs
out.docx

C:\Users\grault\Desktop\a>

Written by grault

May 15, 2009 - 9:46 am at May 15, 2009 - 9:46 am

Posted in notes, script, windowns

command line gmail; msmtp; 465

leave a comment »

Just a few words about my attempt to use Gnus with msmtp. The first step was to launch msmtp without Gnus from command line. There are three ports related to smtp protocol: 25, 465, 587. Usually 25 is the plain smtp without encryption, the difference between the other two is that 587 uses TLS encryption and 465 uses SSL encryption. 587 is filtered in some way on my network (telnet to that port times out), so 465 is the one I use. The majority of examples and tutorials use 587, so I tried using those config files and steps but the port. The most important difference is in the .msmtprc file:

account gmail
host smtp.gmail.com
tls on
tls_certcheck off
tls_starttls off # this is the most important difference
port 465 # not using 587
auth on
user username # without the 'at gmail dot com'
from username@gmail.com
password your_password_connected_to_username

 

Without this most important change msmtp just hangs after printing out “reading recipients from the command line” which is really annoying because there’s no good search results on google what to do in such situations. With previous config file I was able to run msmtp as follows:

$ cat mail.txt
To: grault@do-not-send-mail-here.com
Subject: test msmtp
From: XY <no-such-user@gmail.com>
User-Agent: The command line MSMTP

test MSMTP

--
BRs,
Grault
$ msmtp -d -C .msmtprc -a gmail grault@do-not-send-mail-here.com < mail.txt
ignoring system configuration file C:\ProgramData\msmtprc.txt: No such file or d
irectory
loaded user configuration file .msmtprc
using account gmail from .msmtprc
host                  = smtp.gmail.com
port                  = 465
timeout               = off
protocol              = smtp
domain                = localhost
auth                  = choose
user                  = no-such-user
password              = *
ntlmdomain            = (not set)
tls                   = on
tls_starttls          = off
tls_trust_file        = (not set)
tls_crl_file          = (not set)
tls_key_file          = (not set)
tls_cert_file         = (not set)
tls_certcheck         = off
tls_force_sslv3       = off
tls_min_dh_prime_bits = (not set)
tls_priorities        = (not set)
auto_from             = off
maildomain            = (not set)
from                  = no-such-user@gmail.com
dsn_notify            = (not set)
dsn_return            = (not set)
keepbcc               = off
logfile               = (not set)
syslog                = (not set)
reading recipients from the command line
TLS certificate information:
    Owner:
        Common Name: smtp.gmail.com
        Organization: Google Inc
        Locality: Mountain View
        State or Province: California
        Country: US
    Issuer:
        Common Name: Thawte Premium Server CA
        Organization: Thawte Consulting cc
        Organizational unit: Certification Services Division
        Locality: Cape Town
        State or Province: Western Cape
        Country: ZA
    Validity:
        Activation time: Mon Jul 30 02:00:00 2007
        Expiration time: Fri Jul 30 01:59:59 2010
    Fingerprints:
        SHA1: 5E:F7:E8:CE:1A:BE:D8:94:F2:77:45:5D:ED:38:46:4F:5D:D1:97:61
        MD5:  F1:D3:DE:59:9D:9C:E2:31:EA:AA:2C:A0:FC:AD:9A:61
<-- 220 mx.google.com ESMTP 37sm7124747yxl.35
--> EHLO localhost
<-- 250-mx.google.com at your service, [187.97.152.29]
<-- 250-SIZE 35651584
<-- 250-8BITMIME
<-- 250-AUTH LOGIN PLAIN
<-- 250-ENHANCEDSTATUSCODES
<-- 250 PIPELINING
--> AUTH PLAIN AVL4YWLvbGNuLnN6sWQzAGWkc2Z8eHN6rg==
<-- 235 2.7.0 Accepted
--> MAIL FROM:<no-such-user@gmail.com>
--> RCPT TO:<grault@do-not-send-mail-here.com>
--> DATA
<-- 250 2.1.0 OK 37sm7124747yxl.35
<-- 250 2.1.5 OK 37sm7124747yxl.35
<-- 354  Go ahead 37sm7124747yxl.35
--> To: grault@do-not-send-mail-here.com
--> Subject: test msmtp
--> From: XY <no-such-user@gmail.com>
--> User-Agent: The command line MSMTP
-->
--> test MSMTP
-->
--> --
--> BRs,
--> Grault
--> .
<-- 250 2.0.0 OK 1240471557 37sm7124747yxl.35
--> QUIT
<-- 221 2.0.0 closing connection 37sm7124747yxl.35
$ msmtp --version
msmtp version 1.4.17
TLS/SSL library: GnuTLS
Authentication library: GNU SASL
Supported authentication methods:
plain cram-md5 digest-md5 external login ntlm
IDN support: enabled
NLS: disabled
Keyring support: none
System configuration file name: C:\ProgramData\msmtprc.txt
User configuration file name: c:\users\grault\msmtprc.txt

Copyright (C) 2008 Martin Lambers and others.
This is free software.  You may redistribute copies of it under the terms of
the GNU General Public License <http://www.gnu.org/licenses/gpl.html>.
There is NO WARRANTY, to the extent permitted by law.
$ pwd
/cygdrive/c/users/grault
$

Written by grault

April 23, 2009 - 8:40 am at April 23, 2009 - 8:40 am

collecting coupons

leave a comment »

I’m playing Mafia Wars on Facebook. By finishing a job, you have the chance to get a pecial item, which is part of one from the many collections. When a collection of yours is complete you can get a reward for it.

The action of getting a special item for a job is supposed to have Geometric distribution. So if the probability of getting a special item during that job is 0.1, then you’ll get special items, on average, after every 10 (1/0.1) jobs.

But what is the expected number of special items you have to collect, on average, to complete your collection? This is the Coupon collector’s problem. A lisp code for it is like this:

(defun e-coupon (m)
  (loop as i from 1 upto m
       sum (/ m
	      (+ 1 (- m i)))))

 

For m=7 (which is the usual number of different types of special items in a given collection) this is 18.15. So I expect one collection to be complete only after 181 jobs completed if previous paragraphs hold.

Written by grault

April 10, 2009 - 4:29 pm at April 10, 2009 - 4:29 pm

Posted in lisp, math

data matrix

leave a comment »

grault

grault

I’ve tried "Data Matrix" technology today with my cellphone and i-nigma reader. It works.

Written by grault

March 25, 2009 - 4:10 pm at March 25, 2009 - 4:10 pm

Posted in noise, notes, random