git – peer repositories
$ mkdir a $ cd a $ git init Initialized empty Git repository in /cygdrive/c/users/grault/a/.git/ $ echo a > a $ git add a $ git commit -m "first" [master (root-commit) 8a4bdd7] first 1 files changed, 1 insertions(+), 0 deletions(-) create mode 100644 a $ cd .. $ mkdir b $ cd b $ git init Initialized empty Git repository in /cygdrive/c/users/grault/b/.git/ $ git remote add a ../a $ git branch -a $ git fetch a remote: Counting objects: 3, done. remote: Total 3 (delta 0), reused 0 (delta 0) Unpacking objects: 100% (3/3), done. From ../a * [new branch] master -> a/master $ git branch -a remotes/a/master $ git merge a/master $ git branch -a * master remotes/a/master $ ls a $ cat a a $ cd .. $ cd a $ git remote add b ../b $ git fetch b From ../b * [new branch] master -> b/master $ git branch -a * master remotes/b/master $
hu.dwim.perec tutorial, lisp clos persistency
Let us see a session which demonstrates the possibility of persisting objects into a database and get it back later on by using project hu.dwim.perec (sometimes available on dwim.hu, when the server is up and running). The slots here are of type integer, but there’s a persistence strategy even for type t, when (AFAIK) some kind of internal representation is dumped. To setup a postgresql database for this session, feel free to read further here.
CL-USER> (setf *load-verbose* nil)
NIL
CL-USER> (setf *compile-verbose* nil)
NIL
CL-USER> (asdf:oos 'asdf:load-op :hu.dwim.perec.postgresql :verbose nil)
#<ASDF:LOAD-OP (:VERBOSE NIL) {B2BC6C9}>
CL-USER> (defpackage :ptest
(:use :hu.dwim.common
:hu.dwim.def
:hu.dwim.defclass-star
:hu.dwim.perec
:hu.dwim.rdbms))
#<PACKAGE "PTEST">
CL-USER> (in-package :ptest)
#<PACKAGE "PTEST">
PTEST> (def special-variable
*psql-db*
(make-instance 'postgresql/perec
:generated-transaction-class-name 'transaction
:default-result-type 'vector
:muffle-warnings t
:connection-specification '(:database "perec_db1"
:user-name "perec_user"
:port 5433
:host "localhost"
:password "perec999pass")))
#<POSTGRESQL/PEREC {B4BF189}>
PTEST> (def persistent-class*
some-user-info ()
((userid :type integer)
(some-data :type integer)))
#<PERSISTENT-CLASS SOME-USER-INFO>
PTEST> (make-compiled-query-cache)
#<HASH-TABLE :TEST EQUAL :COUNT 0 {C53BBC1}>
PTEST> (setf hu.dwim.perec::*compiled-query-cache* *)
#<HASH-TABLE :TEST EQUAL :COUNT 0 {C53BBC1}>
PTEST> (setf (hu.dwim.logger:log-level 'hu.dwim.rdbms::rdbms)
hu.dwim.logger:+fatal+
(hu.dwim.logger:log-level 'hu.dwim.rdbms::sql)
hu.dwim.logger:+fatal+)
5
PTEST> (with-database *psql-db*
(with-transaction
(make-instance 'some-user-info :userid 1 :some-data -1)))
#<SOME-USER-INFO :persistent #t 12>
PTEST> (with-database *psql-db*
(with-transaction
(make-instance 'some-user-info :userid 2 :some-data -2)))
#<SOME-USER-INFO :persistent #t 13>
PTEST> (with-database *psql-db*
(with-transaction
(select (o)
(from (o some-user-info))
(where (= 1 (userid-of o))))))
(#<SOME-USER-INFO :persistent #? 12 {B14A2E1}>)
PTEST> (with-database *psql-db*
(with-transaction
(some-data-of (revive-instance (car *)))))
-1
PTEST>
Tables are created automatically in the appropriate database:
perec_db1=# select * from _some_user_info;
_oid | _userid | _some_data
---------+---------+------------
960005 | 1 | -1
1025541 | 2 | -2
(2 rows)
perec_db1=# \d _some_user_info
Table "public._some_user_info"
Column | Type | Modifiers
------------+---------+-----------
_oid | bigint | not null
_userid | numeric |
_some_data | numeric |
Indexes:
"_some_user_info_pkey" PRIMARY KEY, btree (_oid)
perec_db1=#
the (not so) mysterious common lisp loop
A good interview question: what is the value of the following form?
CL-USER> (mapcar #'funcall (loop as i from 1 upto 3 collect (lambda () i))) ???
“The Truth Is Out There” where macroexpand lives.
CL-USER> (macroexpand `(loop for i from 1 upto 3 collect (lambda () i)))
(BLOCK NIL
(LET ((I 1))
(DECLARE (TYPE (AND REAL NUMBER) I))
(SB-LOOP::WITH-LOOP-LIST-COLLECTION-HEAD (#:LOOP-LIST-HEAD-1070
#:LOOP-LIST-TAIL-1071)
(SB-LOOP::LOOP-BODY NIL
(NIL NIL (WHEN (> I '3) (GO SB-LOOP::END-LOOP)) NIL)
((SB-LOOP::LOOP-COLLECT-RPLACD
(#:LOOP-LIST-HEAD-1070 #:LOOP-LIST-TAIL-1071)
(LIST (LAMBDA () I))))
(NIL (SB-LOOP::LOOP-REALLY-DESETQ I (1+ I))
(WHEN (> I '3) (GO SB-LOOP::END-LOOP)) NIL)
((RETURN-FROM NIL
(SB-LOOP::LOOP-COLLECT-ANSWER
#:LOOP-LIST-HEAD-1070)))))))
T
CL-USER>
Therefore i is a variable and different values are bound to it during the loop. This is explained very well here.
The good candidate answers (4 4 4), because it’s clearly stated on previous page, how this type of loop terminates.
join variable length multiline data entries with sed
% cat test.txt data1-1 data1-2 data1-3 closing-form data2-1 data2-2 data2-3 data2-4 closing-form data3-1 data3-2 closing-form % cat test.txt | sed -n -e ':x;N;/\nclosing-form$/!bx;s/\n/;/g;p' data1-1;data1-2;data1-3;closing-form data2-1;data2-2;data2-3;data2-4;closing-form data3-1;data3-2;closing-form %
A precodition is that there’s no empty data entry. If there is, you can introduce helper entries:
% cat testt.txt data1-1 data1-2 closing-form closing-form closing-form data3-1 data3-2 closing-form % cat testt.txt | sed '/^closing-form$/ihelper-entry' data1-1 data1-2 helper-entry closing-form helper-entry closing-form helper-entry closing-form data3-1 data3-2 helper-entry closing-form %
testing, states, comparison
Repeatable test means that you can build up the _same_ starting state and run the _same_ test cases with the _same_ test tools configured the _same_ way. Even if you have random simulations, you should be able to reset the random seed.
It can happen that you cannot use the same starting state in your project, because it is around a terabyte (a database containing transactions and their history). You can’t step twice into the same database : ) Some sort of justification can be that this big state is an aggregate state. So a given test case depends only on its close environment. In this case though, the locality of these environments should be documented.
It can happen that the schema of the database changes and because of this, some of the numerous test cases are modified during the regression tests. But not all the ones that depend on the old schema (or on the schema in general). Years later nobody knows what the hell is going on when testing with those old cases. Either about the way they should be fixed.
By the way, don’t represent the cases in MS Access databases. In case you want to understand the evolution of a test case (e.g. along with the evolution of the database schema) by looking through the versioned history of their representation, that’s quite time consuming because there aren’t any good tools to diff MS Access databases.
I’m not sure though, that teaching people comparable textual representations worth it in contrast to not comparable wysiwyg editor based information forms. A solution can be a wysiwyg solution with predictable and _not_ verbose export functionality. For example pg_dump and pgadmin as GUI:
% pg_dump -s | grep -v ^-- > dbscm1.sql % psql -q => create table zfoo (a serial primary key, c numeric check (c > 0)); NOTICE: CREATE TABLE will create implicit sequence "zfoo_a_seq" for "serial" column "zfoo.a" NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "zfoo_pkey" for table "zfoo" => \q % pg_dump -s | grep -v ^-- > dbscm2.sql % diff dbscm1.sql dbscm2.sql 270a271,278 > CREATE TABLE zfoo ( > a serial NOT NULL, > c numeric, > CONSTRAINT zfoo_c CHECK ((c > (0)::numeric)) > ); > > > 340a349,353 > ALTER TABLE ONLY zfoo > ADD CONSTRAINT zfoo_pkey PRIMARY KEY (a); > > > %
graph algorithms with GraphViz and agraph
% find . ./bin ./bin/cdt.dll ./bin/graph.dll ./bin/sample.gv ./incomings.c ./Makefile %
These are the files from this hello world project. The dlls are from the GraphViz installation directory (from bin/). The task is to collect incoming nodes for a given other node (the ones that have directed edges into the selected one). The good thing in this is that you don’t have to design you own graph markup language and your own parser (NIH). Instead you can use the well defined and supported dot language from GraphViz. I’ve used MingW and MSYS as development environment.
% cat -A Makefile
all:^Iincomings.exe$
$
incomings.exe:^Iincomings.o$
^Ig++ $< -O2 -o bin/$@ `PKG_CONFIG_PATH=c:/Graphviz2.24/lib/pkgconfig/ pkg-confi
g --libs libgvc`$
$
incomings.o:^Iincomings.c$
^Ig++ $< -O2 -c -o $@ `PKG_CONFIG_PATH=c:/Graphviz2.24/lib/pkgconfig/ pkg-config
--cflags libgvc`$
$
clean:$
^Irm -rf incomings.o bin/incomings.exe$
$
test:^Iall$
^Ibin/incomings.exe bin/sample.gv a$
% cat incomings.c
#include<gvc.h>
char* gv_gets(char *buf, int n, FILE* fp)
{
return fgets(buf,n,fp);
}
int main (int argc, char **argv)
{
Agraph_t *g;
Agnode_t *n;
Agedge_t *e;
FILE *fp;
if(argc > 2)
fp=fopen(argv[1],"r");
else
fp=stdin;
aginit();
g=agread_usergets(fp,gv_gets);
for(n=agfstnode(g);n;n=agnxtnode(g,n))
if(strcmp(n->name,argv[argc-1])==0)
for(e=agfstin(g,n);e;e=agnxtin(g,e))
printf("%s\n",e->tail->name);
agclose(g);
return 0;
}
% make
g++ incomings.c -O2 -c -o incomings.o `PKG_CONFIG_PATH=c:/Graphviz2.24/lib/pkgco
nfig/ pkg-config --cflags libgvc`
g++ incomings.o -O2 -o bin/incomings.exe `PKG_CONFIG_PATH=c:/Graphviz2.24/lib/pk
gconfig/ pkg-config --libs libgvc`
% cat bin/sample.gv
digraph G
{
b -> a;
c -> a;
d -> a;
e -> a;
f -> a;
a -> foo;
}
% make test
g++ incomings.o -O2 -o bin/incomings.exe `PKG_CONFIG_PATH=c:/Graphviz2.24/lib/pk
gconfig/ pkg-config --libs libgvc`
bin/incomings.exe bin/sample.gv a
b
c
d
e
f
%
granting a loan; simplified
Based on the net present value term, in case of regular $R incomes and rate as discount rate:
years - 1
---------
\
\ R
NPV = ) -------------
/ i
/ / \
--------- | 1 + rate |
i = 0 \ /
Due to identical cash flows this is the sum of a geometric series, so can be rewritten into a closed form:
years
/ \
| 1 |
1 - | -------- |
| 1 + rate |
\ /
NPV = R ---------------------
1
1 - --------
1 + rate
Or identically:
1
1 - --------
1 + rate
R = NPV ---------------------
years
/ \
| 1 |
1 - | -------- |
| 1 + rate |
\ /
In case duration of the loan tends to infinity then the required yearly income converges to:
/ \
| 1 |
NPV | 1 - -------- |
| 1 + rate |
\ /
which is
rate
NPV --------
1 + rate
and this means that there’s no loan of USD NPV regarding rate as discount rate, where the required yearly pay back is less then this number, e.g. (NPV = $1000000, rate = 0.03):
% ./lx86cl Welcome to Clozure Common Lisp Version 1.4-r13119 (LinuxX8632)! ? (defun limit (npv rate) (* npv (/ rate (+ 1 rate)))) LIMIT ? (limit 1000000 0.03) 29126.213 ? (defun r (npv rate years) (* npv (/ (- 1 (/ 1 (+ 1 rate))) (- 1 (expt (/ 1 (+ 1 rate)) years))))) R ? (loop as i from 1 upto 50 collect (r 1000000 0.03 i)) (1000000.0 507389.3 343233.47 261191.42 211994.77 179220.95 155831.47 138307.22 124693.086 113816.05 104929.58 97536.03 91290.83 85947.914 81326.79 77292.09 73740.32 70590.97 67780.47 65257.973 62982.297 60919.8 59042.617 57327.58 55755.207 54309.016 52974.95 51740.992 50596.758 49533.246 48542.637 47618.066 46753.5 45943.64 45183.758 44469.687 43797.676 43164.387 42566.824 42002.29 41468.336 40962.766 40483.58 40028.957 39597.234 39186.906 38796.586 38425.0 38070.984 37733.465) ? (with-open-file (s #P"r.dat" :direction :output :if-exists :supersede) (loop as val in * do (format s "~A~%" val))) NIL ? (quit) % echo set term png \; plot \'r.dat\' with lines | gnuplot > r.png
r.png
rebuild cffi foreign bindings
C:\grault>type lib-foo.c
int getone ()
{
return 1;
}
C:\grault>gcc -shared -o lib-foo.dll lib-foo.c
C:\grault>clisp -q -norc -i ..\.clisprc
;; Loading file ..\.clisprc ...
[1]> (setf *load-verbose* nil)
NIL
[2]> (asdf:oos 'asdf:load-op :cffi :verbose nil)
#<ASDF:LOAD-OP (:VERBOSE NIL) #x19C29B55>
[3]> (cffi:load-foreign-library "lib-foo")
#<CFFI::FOREIGN-LIBRARY #x19CCE065>
[4]> (cffi:defcfun "getone" :int)
GETONE
[5]> (getone)
1
[6]> (ext:saveinitmem "foo.mem" :norc t)
Bytes permanently allocated: 92,512
Bytes currently in use: 3,409,664
Bytes available until next GC: 852,166
3409664 ;
852166 ;
92512 ;
38 ;
20303572 ;
2808018
[7]> (quit)
C:\grault>dir/b
foo.mem
lib-foo.c
lib-foo.dll
C:\grault>clisp -q -M foo.mem
[1]> (getone)
WARNING: FFI::FIND-FOREIGN-FUNCTION: no dynamic object named "getone" in library :DEFAULT
*** - FUNCALL: undefined function NIL
The following restarts are available:
USE-VALUE :R1 Input a value to be used instead of (FDEFINITION 'NIL).
RETRY :R2 Retry
STORE-VALUE :R3 Input a new value for (FDEFINITION 'NIL).
ABORT :R4 Abort main loop
Break 1 [2]> :r4
[3]> (cffi:use-foreign-library "lib-foo")
#<CFFI::FOREIGN-LIBRARY #x19FB66AD>
[4]> (getone)
1
[5]> (quit)
C:\grault>
common lisp GUI-application shipment&delivery on windows
First of all, here’s the lisp I’ve used:
% clisp --version | grep -v ^Machine GNU CLISP 2.48 (2009-07-28) (built on stnt067 [192.168.0.1]) Software: GNU C 3.4.5 (mingw-vista special r3) gcc -mno-cygwin -O2 -W -Wswitch -Wcomment -Wpointer-arith -Wimplicit -Wreturn-ty pe -Wmissing-declarations -Wno-sign-compare -Wno-format-nonliteral -O2 -fexpensi ve-optimizations -falign-functions=4 -D_WIN32 -DUNICODE -DDYNAMIC_FFI -I. -lint l -lreadline -ltermcap -lavcall -lcallback -luser32 -lws2_32 -lole32 -loleaut32 -luuid -liconv -lsigsegv SAFETY=0 HEAPCODES STANDARD_HEAPCODES GENERATIONAL_GC SPVW_BLOCKS SPVW_MIXED TRI VIALMAP_MEMORY libsigsegv 2.6 libiconv 1.11 libreadline 5.2 Features: (READLINE REGEXP SYSCALLS I18N LOOP COMPILER CLOS MOP CLISP ANSI-CL COMMON-LISP LISP=CL INTERPRETER SOCKETS GENERIC-STREAMS LOGICAL-PATHNAMES SCREEN FFI GETTEXT UNICODE BASE-CHAR=CHARACTER PC386 WIN32) C Modules: (clisp i18n syscalls regexp readline) Installation directory: C:\Program Files\clisp-2.48\ User language: ENGLISH %
I have also MingW (5.1.6) and MSYS (1.0.1) installed, Witchs Hat icon downloaded, CLisp dlls copied from the base subdirectory under the installation path. This is the development tree in clean form:
% find . ./bootstrapper.cc ./bootstrapper.rc ./clisp-dlls ./clisp-dlls/libiconv-2.dll ./clisp-dlls/libintl-8.dll ./clisp-dlls/readline5.dll ./Makefile ./message.lisp ./witchs-hat.ico %
Under clisp-dlls, there are the mentioned dlls of CLisp. The file message.lisp contains the program you’d like to run (or ship or whatever). Finally, the bootstrapper files are up to provide an exe with company info, icon, copyright, etc. as an entry to the lisp program. The important bit here is that you don’t want a command prompt window popping up before startup.
Let’s see the files:
% cat message.lisp
(use-package "FFI")
(def-call-out messagebox
(:name "MessageBoxA") (:library "user32.dll")
(:arguments (hwnd int) (text c-string) (capt c-string) (type uint))
(:return-type int)
(:language :stdc))
(defun main ()
(messagebox 0 "Your hacking starts... NOW!" "Demo MsgBox" 0)
(quit))
% cat -A Makefile
all:^Imsgbox-app.exe msgbox-app.img$
^Imkdir -p shipment$
^Icp msgbox-app.exe shipment$
^Icp msgbox-app.img shipment$
^Icp clisp-dlls/* shipment$
$
msgbox-app.img:^Imsgbox-app.img.exe$
^Icp $< $@$
$
msgbox-app.img.exe:^Imessage.lisp$
^Iclisp -q -norc -x \$
"(load \"message.lisp\") \$
(ext:saveinitmem #P\"./msgbox-app.img.exe\" \$
:executable t \$
:norc t \$
:init-function #'main)"$
$
bootstrapper.res:^Ibootstrapper.rc witchs-hat.ico$
^Iwindres $< -O coff -o $@$
^I$
msgbox-app.exe:^Ibootstrapper.cc bootstrapper.res$
^Ig++ -mwindows $^ -o $@$
$
clean:$
^Irm -rf msgbox-app.exe bootstrapper.res \$
^Imsgbox-app.img msgbox-app.img.exe shipment$
% cat bootstrapper.rc
ID ICON "witchs-hat.ico"
1 VERSIONINFO
FILEVERSION 1,0,0,0
PRODUCTVERSION 1,0,0,0
BEGIN
BLOCK "StringFileInfo"
BEGIN
BLOCK "080904E4"
BEGIN
VALUE "CompanyName", "MsgBox Products"
VALUE "FileDescription", "Demo MsgBox"
VALUE "FileVersion", "1.0"
VALUE "InternalName", "msgbox-app"
VALUE "LegalCopyright", "Grault"
VALUE "OriginalFilename", "msgbox-app.exe"
VALUE "ProductName", "Demo MsgBox"
VALUE "ProductVersion", "1.0"
END
END
BLOCK "VarFileInfo"
BEGIN
VALUE "Translation", 0x809, 1252
END
END
% cat bootstrapper.cc
#include <windows.h>
int WinMain(HINSTANCE a0, HINSTANCE a1, LPSTR a2, int a3)
{
STARTUPINFO si;
PROCESS_INFORMATION pi;
ZeroMemory(&si,sizeof(STARTUPINFO));
si.cb = sizeof(STARTUPINFO);
CreateProcess("msgbox-app.img", "",
NULL,NULL,TRUE,CREATE_NO_WINDOW,NULL,
NULL,&si,&pi);
// version and company details are in the bootstrapper exe
// it's kinda better to show these e.g. in process explorer
// and image file as a subprocess
// feel free to comment the following line to leave img only
WaitForSingleObject(pi.hProcess, INFINITE);
return 0;
}
%
These are the files, and here are the results of a make:
% make
windres bootstrapper.rc -O coff -o bootstrapper.res
g++ -mwindows bootstrapper.cc bootstrapper.res -o msgbox-app.exe
clisp -q -norc -x \
"(load \"message.lisp\") \
(ext:saveinitmem #P\"./msgbox-app.img.exe\" \
:executable t \
:norc t \
:init-function #'main)"
;; Loading file message.lisp ...
;; Loaded file message.lisp
T
;; Wrote the memory image into .\msgbox-app.img.exe (5,088,157 bytes)
Bytes permanently allocated: 92,512
Bytes currently in use: 2,188,704
Bytes available until next GC: 544,546
2188704 ;
544546 ;
92512 ;
1 ;
52160 ;
156001
cp msgbox-app.img.exe msgbox-app.img
mkdir -p shipment
cp msgbox-app.exe shipment
cp msgbox-app.img shipment
cp clisp-dlls/* shipment
% find
.
./bootstrapper.cc
./bootstrapper.rc
./bootstrapper.res
./clisp-dlls
./clisp-dlls/libiconv-2.dll
./clisp-dlls/libintl-8.dll
./clisp-dlls/readline5.dll
./Makefile
./message.lisp
./msgbox-app.exe
./msgbox-app.img
./msgbox-app.img.exe
./shipment
./shipment/libiconv-2.dll
./shipment/libintl-8.dll
./shipment/msgbox-app.exe
./shipment/msgbox-app.img
./shipment/readline5.dll
./witchs-hat.ico
Here’s a picture how this all looks like on a Windows Server 2003 Standard Edition without any kind of lisp installed. And the entries in Process Explorer.
Some sort of shipment : )
show up in Process Explorer
Obviously one can develop a much more sophisticated bootstrapper. This one represents only the possibility of releasing software written in Lisp. Assembling these output files together into an installer could be the next step : ))
And finally, you can get either the binaries or the source files by sending a mail to:
% echo gra.rtebsrpehbf.ferfh@gyhnet | rev | tr '[a-z]' '[n-za-m]'
7zip loads of files in a single folder
I’ve met with a folder with around 400000 files on Windows. Listing the files takes more than 5 minutes for an explorer window which is infinity in IT terms : ) Setting the working directory to this one in a cmd window works. Great. Starting a dir command in cmd leads to this infinity of silence also. Suprisingly if your file names are structured enough to hash them with a prefix, then `dir prefix1*’ is relatively fast. Therefore creating directories for the values and moving the groups to these solves the problem somewhat. I also decided to zip these folders of grouped files (by a hash on their names). 7z run time somehow depends on the nr of files in current directory even if you’re up to compress the files in a single subdirectory into a file which is in the current working one. This can be an issue when you have hundreds of thousands files in a single directory. So it’s better to separate the files first and compress them after.
Here’s the batch file I’ve used to solve this.
set list=^ 200801 200802 200803 200804 200805 200806 ^ 200807 200808 200809 200810 200811 200812 ^ 200901 200902 200903 200904 200905 200906 ^ 200907 200908 200909 for %%i in (%list%) do ( md archive_%%i move someprefix%%i* archive_%%i\ ) for %%i in (%list%) do ( 7z a archive_%%i.7z archive_%%i\ )
