join & filter multiline data records with sed
Grouping with sed:
$ cat nrs.txt 01 02 03 04 05 06 07 08 09 10 11 12 $ cat nrs.txt | sed -n -e 'N;s/\n/ /g;p' 01 02 03 04 05 06 07 08 09 10 11 12 $ cat nrs.txt | sed -n -e 'N;N;s/\n/ /g;p' 01 02 03 04 05 06 07 08 09 10 11 12 $ cat nrs.txt | sed -n -e 'N;N;N;s/\n/ /g;p' 01 02 03 04 05 06 07 08 09 10 11 12 $ cat nrs.txt | sed -n -e 'N;N;N;N;N;s/\n/ /g;p' 01 02 03 04 05 06 07 08 09 10 11 12 $
Good for the following task:
$ cat entries.txt entry-1-data-1 entry-1-data-2 entry-2-data-1 entry-2-data-2 $ cat entries.txt | sed -n -e 'N;s/\n/ /;p' entry-1-data-1 entry-1-data-2 entry-2-data-1 entry-2-data-2 $
Two long groups, removing elements (1st, 2nd):
$ cat nrs.txt | sed -n -e 'p;n' 01 03 05 07 09 11 $ cat nrs.txt | sed -n -e 'n;p' 02 04 06 08 10 12 $
In general case you have groups of length _k_ and the starting pattern is ‘n;n;…;n’. There’s k-1 number of letter n here. You can place letter p around the n-s, which is exactly k possibility. If there’s a p in position k0 then the k0-th element will be printed out.
So if you have 6 long groups and you want every fifth element:
$ cat nrs.txt | sed -n -e 'n;n;n;n;p;n' 05 11 $
office open xml file sample
Here’s the spec.
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<?mso-application progid="Word.Document"?>
<pkg:package xmlns:pkg="http://schemas.microsoft.com/office/2006/xmlPackage">
<pkg:part
pkg:name="/_rels/.rels"
pkg:contentType="application/vnd.openxmlformats-package.relationships+xml">
<pkg:xmlData>
<Relationships
xmlns="http://schemas.openxmlformats.org/package/2006/relationships">
<Relationship
Id="foo"
Type=
"http://schemas.openxmlformats.org/officeDocument/2006/relationships/officeDocument"
Target="/word/document.xml"/>
</Relationships>
</pkg:xmlData>
</pkg:part>
<pkg:part
pkg:name="/word/document.xml"
pkg:contentType=
"application/vnd.openxmlformats-officedocument.wordprocessingml.document.main+xml">
<pkg:xmlData>
<w:document
xmlns:w="http://schemas.openxmlformats.org/wordprocessingml/2006/main">
<w:body>
<w:p>
<w:r>
<w:t>Hello World</w:t>
<w:br />
<w:yearLong />
</w:r>
</w:p>
</w:body>
</w:document>
</pkg:xmlData>
</pkg:part>
</pkg:package>
cumulating minutes begun
$ cat seconds.txt 120 123 $ cat seconds.txt | sed 's/$/ 60 ~ 0 !=r +/' | sed '1i[1+] sr 0' | sed '$ap' | dc 5 $
compiler with flex & bison in 77 lines
Please note that you cannot copy-paste the Makefile below.
afroid-laptop% ls
lexer.l Makefile parser.y
afroid-laptop% cat * | wc -l
77
afroid-laptop% cat -A Makefile
all:^Icompiler$
$
compiler:^Iparser.tab.o lex.yy.o$
^Igcc -o $@ parser.tab.o lex.yy.o -lm -lfl$
$
parser.tab.o:^Iparser.y$
^Ibison -dv parser.y$
^Igcc -c parser.tab.c$
$
lex.yy.o:^Ilexer.l parser.tab.o$
^Iflex lexer.l$
^Igcc -c lex.yy.c$
$
clean:$
^Irm -rf lex.yy.c lex.yy.o parser.tab.o parser.tab.c parser.tab.h parser.output compiler$
afroid-laptop% cat lexer.l
%{
#include
#include "parser.tab.h"
%}
DIGIT [0-9]
ALPHA [a-zA-Z]
WS [ \n\t]
%%
{DIGIT}+ { yylval.intval = atoi(yytext); return NUMBER; }
{ALPHA}+ { yylval.id = (char *) strdup(yytext); return IDENTIFIER; }
{WS}+ {}
%%
int yywrap (void)
{
}
afroid-laptop% cat parser.y
%{
#include
#include
extern FILE *yyin;
extern FILE *yyout;
%}
%union
{
int intval;
char* id;
}
%start S
%token NUMBER
%token IDENTIFIER
%%
S:
/* empty */
| IDENTIFIER S NUMBER { fprintf (yyout, "let %s = %d;\n", $1, $3); }
%%
main (int argc, char **argv)
{
char out[1024];
++argv; --argc;
yyin = fopen (argv[0], "r");
strcpy(out,argv[0]);
strcat(out,".out");
yyout = fopen (out, "w");
yyparse ();
printf ("Parse Completed\n");
}
yyerror (char *s)
{
printf("error\n");
}
afroid-laptop% cat > test.src << EOF
heredoc> a b c 1 2 3
heredoc> EOF
afroid-laptop% make
bison -dv parser.y
gcc -c parser.tab.c
flex lexer.l
gcc -c lex.yy.c
gcc -o compiler parser.tab.o lex.yy.o -lm -lfl
afroid-laptop% ./compiler test.src
Parse Completed
afroid-laptop% cat test.src.out
let c = 1;
let b = 2;
let a = 3;
afroid-laptop%
core context oriented programming with ContextL
ContextL is a part of the Closer project with contributions from Pascal Costanza and probably others. As per the code repository, it’s actively developed.
(defpackage :a-test
(:use :cl))
(in-package :a-test)
(contextl:define-layered-class
bar ()
((value :initarg :value :layered-accessor bar-value)))
(contextl:define-layered-function bar-incf (f))
(contextl:define-layered-method
bar-incf ((b bar))
(incf (bar-value b)))
(contextl:deflayer log-layer)
(contextl:define-layered-class
bar :in log-layer ()
())
(contextl:define-layered-method
bar-incf :in log-layer :around ((b bar))
(progn
(princ (format nil "~A is incfed" b))
(call-next-method)))
(defparameter +a-bar+
(make-instance 'bar :value 11))
And now, the REPL:
A-TEST> +a-bar+
#<BAR {BEB2581}>
A-TEST> (bar-value +a-bar+)
11
A-TEST> (bar-incf +a-bar+)
12
A-TEST> (bar-value +a-bar+)
12
A-TEST> (contextl:with-active-layers (log-layer) (bar-incf +a-bar+))
#<BAR {BEB2581}> is incfed
13
A-TEST> (bar-value +a-bar+)
13
A-TEST>
ms access relations into graphviz
Today is a scripting day : )
If you have to maintain a large MS Access database (manipulate data in it) with complex relationship matrix, the built-in relations diagram is quite useless. This is because Access won’t compute a proper layout. It just displays the nodes (the tables) next to each other and draw all the edges. Period. An idea is to dump these relations into a graphviz .dot file and let graphviz do the layout. This solution is very useful if you want to understand a complex database represented in Access.
C:\Users\grault\Desktop>dir/b test.accdb
test.accdb
C:\Users\grault\Desktop>type dumpRelationship.vbs
Set objFSO = CreateObject("Scripting.FileSystemObject")
Set objAcc = CreateObject("Access.Application")
fileName = objFSO.GetAbsolutePathName(WScript.Arguments.Item(0))
outName = objFSO.GetAbsolutePathName(WScript.Arguments.Item(1))
objAcc.OpenCurrentDatabase(fileName)
Set outFile = objFSO.CreateTextFile(outName)
outFile.WriteLine ("digraph G { graph [rankdir=""LR""];")
For Each tbl In objAcc.CurrentDb.Tabledefs
If LCase(Left(tbl.Name, 4)) <> "msys" Then
entry = """" & tbl.Name & """ [label=""<" & tbl.Name & "> " & tbl.Name
For Each fi In tbl.Fields
entry = entry & "| <" & fi.Name & "> " & fi.Name
Next
entry = entry & """ shape=""record""];"
outFile.WriteLine(entry)
End If
Next
For Each rel In objAcc.CurrentDb.Relations
For Each fld In rel.Fields
arrow = """" & rel.Table & """:" & fld.Name & " -> " & _
"""" & rel.ForeignTable & """:" & fld.ForeignName & _
" [arrowhead=""odotodot"" arrowtail=""orboxrtee""];"
outFile.WriteLine(arrow)
Next
Next
outFile.WriteLine("}")
outFile.Close
objAcc.CloseCurrentDataBase
objAcc.Quit
C:\Users\grault\Desktop>dumpRelationship.vbs test.accdb test.dot
C:\Users\grault\Desktop>type test.dot
digraph G { graph [rankdir="LR"];
"Table1" [label="<Table1> Table1| <ID> ID| <Desc> Desc" shape="record"];
"Table2" [label="<Table2> Table2| <ID> ID| <Ref1> Ref1| <OtherDesc> OtherDesc"
shape="record"];
"Table1":ID -> "Table2":Ref1 [arrowhead="odotodot" arrowtail="orboxrtee"];
}
C:\Users\grault\Desktop>dot -Tpng -o test.png test.dot
(dot.exe:6480): Pango-WARNING **: couldn't load font
"Times Roman Not-Rotated 14 ",
falling back to "Sans Not-Rotated 14", expect ugly output.
C:\Users\grault\Desktop>
the result
concatenate or merge docx Word files dot vbs
In case you have generated Word files which are parts of a future “master” document and also you want to automate the process of assembling them together by writing some sort of build script, the following snippet is probably useful. Generating documents is good to avoid redundancy. In case you have to have a document in which parts can be computed based on other parts (with also the miserable contraint on document format and production environment) , it’s good to consider splitting the document, generating the parts and assembling them at the end. And now, the code:
C:\Users\grault\Desktop\a>dir/b
a1.docx
a2.docx
build.vbs
C:\Users\grault\Desktop\a>type build.vbs
Set objWord = CreateObject("Word.Application")
Set objFSO = CreateObject("Scripting.FileSystemObject")
Set objDoc = objWord.Documents.Add()
objDoc.ActiveWindow.View.Type = 2 'wdOutlineView
For I = 1 To WScript.Arguments.Count - 1
objDoc.Subdocuments.AddFromFile _
(objFSO.GetAbsolutePathName(WScript.Arguments.Item(I)))
Next
objDoc.SaveAs(objFSO.GetAbsolutePathName(WScript.Arguments.Item(0)))
objWord.Quit
C:\Users\grault\Desktop\a>build.vbs out.docx a1.docx a2.docx
C:\Users\grault\Desktop\a>dir/b
a1.docx
a2.docx
build.vbs
out.docx
C:\Users\grault\Desktop\a>
command line gmail; msmtp; 465
Just a few words about my attempt to use Gnus with msmtp. The first step was to launch msmtp without Gnus from command line. There are three ports related to smtp protocol: 25, 465, 587. Usually 25 is the plain smtp without encryption, the difference between the other two is that 587 uses TLS encryption and 465 uses SSL encryption. 587 is filtered in some way on my network (telnet to that port times out), so 465 is the one I use. The majority of examples and tutorials use 587, so I tried using those config files and steps but the port. The most important difference is in the .msmtprc file:
account gmail host smtp.gmail.com tls on tls_certcheck off tls_starttls off # this is the most important difference port 465 # not using 587 auth on user username # without the 'at gmail dot com' from username@gmail.com password your_password_connected_to_username
Without this most important change msmtp just hangs after printing out “reading recipients from the command line” which is really annoying because there’s no good search results on google what to do in such situations. With previous config file I was able to run msmtp as follows:
$ cat mail.txt
To: grault@do-not-send-mail-here.com
Subject: test msmtp
From: XY <no-such-user@gmail.com>
User-Agent: The command line MSMTP
test MSMTP
--
BRs,
Grault
$ msmtp -d -C .msmtprc -a gmail grault@do-not-send-mail-here.com < mail.txt
ignoring system configuration file C:\ProgramData\msmtprc.txt: No such file or d
irectory
loaded user configuration file .msmtprc
using account gmail from .msmtprc
host = smtp.gmail.com
port = 465
timeout = off
protocol = smtp
domain = localhost
auth = choose
user = no-such-user
password = *
ntlmdomain = (not set)
tls = on
tls_starttls = off
tls_trust_file = (not set)
tls_crl_file = (not set)
tls_key_file = (not set)
tls_cert_file = (not set)
tls_certcheck = off
tls_force_sslv3 = off
tls_min_dh_prime_bits = (not set)
tls_priorities = (not set)
auto_from = off
maildomain = (not set)
from = no-such-user@gmail.com
dsn_notify = (not set)
dsn_return = (not set)
keepbcc = off
logfile = (not set)
syslog = (not set)
reading recipients from the command line
TLS certificate information:
Owner:
Common Name: smtp.gmail.com
Organization: Google Inc
Locality: Mountain View
State or Province: California
Country: US
Issuer:
Common Name: Thawte Premium Server CA
Organization: Thawte Consulting cc
Organizational unit: Certification Services Division
Locality: Cape Town
State or Province: Western Cape
Country: ZA
Validity:
Activation time: Mon Jul 30 02:00:00 2007
Expiration time: Fri Jul 30 01:59:59 2010
Fingerprints:
SHA1: 5E:F7:E8:CE:1A:BE:D8:94:F2:77:45:5D:ED:38:46:4F:5D:D1:97:61
MD5: F1:D3:DE:59:9D:9C:E2:31:EA:AA:2C:A0:FC:AD:9A:61
<-- 220 mx.google.com ESMTP 37sm7124747yxl.35
--> EHLO localhost
<-- 250-mx.google.com at your service, [187.97.152.29]
<-- 250-SIZE 35651584
<-- 250-8BITMIME
<-- 250-AUTH LOGIN PLAIN
<-- 250-ENHANCEDSTATUSCODES
<-- 250 PIPELINING
--> AUTH PLAIN AVL4YWLvbGNuLnN6sWQzAGWkc2Z8eHN6rg==
<-- 235 2.7.0 Accepted
--> MAIL FROM:<no-such-user@gmail.com>
--> RCPT TO:<grault@do-not-send-mail-here.com>
--> DATA
<-- 250 2.1.0 OK 37sm7124747yxl.35
<-- 250 2.1.5 OK 37sm7124747yxl.35
<-- 354 Go ahead 37sm7124747yxl.35
--> To: grault@do-not-send-mail-here.com
--> Subject: test msmtp
--> From: XY <no-such-user@gmail.com>
--> User-Agent: The command line MSMTP
-->
--> test MSMTP
-->
--> --
--> BRs,
--> Grault
--> .
<-- 250 2.0.0 OK 1240471557 37sm7124747yxl.35
--> QUIT
<-- 221 2.0.0 closing connection 37sm7124747yxl.35
$ msmtp --version
msmtp version 1.4.17
TLS/SSL library: GnuTLS
Authentication library: GNU SASL
Supported authentication methods:
plain cram-md5 digest-md5 external login ntlm
IDN support: enabled
NLS: disabled
Keyring support: none
System configuration file name: C:\ProgramData\msmtprc.txt
User configuration file name: c:\users\grault\msmtprc.txt
Copyright (C) 2008 Martin Lambers and others.
This is free software. You may redistribute copies of it under the terms of
the GNU General Public License <http://www.gnu.org/licenses/gpl.html>.
There is NO WARRANTY, to the extent permitted by law.
$ pwd
/cygdrive/c/users/grault
$
collecting coupons
I’m playing Mafia Wars on Facebook. By finishing a job, you have the chance to get a pecial item, which is part of one from the many collections. When a collection of yours is complete you can get a reward for it.
The action of getting a special item for a job is supposed to have Geometric distribution. So if the probability of getting a special item during that job is 0.1, then you’ll get special items, on average, after every 10 (1/0.1) jobs.
But what is the expected number of special items you have to collect, on average, to complete your collection? This is the Coupon collector’s problem. A lisp code for it is like this:
(defun e-coupon (m)
(loop as i from 1 upto m
sum (/ m
(+ 1 (- m i)))))
For m=7 (which is the usual number of different types of special items in a given collection) this is 18.15. So I expect one collection to be complete only after 181 jobs completed if previous paragraphs hold.
data matrix
grault
I’ve tried "Data Matrix" technology today with my cellphone and i-nigma reader. It works.
