19 2Gb issue; file distribution
::::::
W
hen a file reaches 2Gb in size it might become corrupted – in case it has j4 type that many T24 files do. The common question is – what to do to have your data safe should it happen?There are several opportunities:
• Convert to some newer type that supports more than 2Gb size. Namely: jp or jr.
Before doing that – think: do you really need so big files in your system? What about backups taking longer and longer time? Most probably you no more need the historical data in that file to be always available? “2 Gb” issue mainly concerns such things like STMT.ENTRY or FUNDS.TRANSFER ($his for the latter) and these files accumulate data that rarely or never change. (You might say that data never changes there – yes, individual records do not but from logical point of view by “rarely” I meant that in FT history an additional record for existing deal might be created later – in case of reversal, for example.)
• Another option is to proceed with “Archive” functionality of T24. It creates a new data file – $arc – and puts there records that belong to dates earlier than a date that you set for archiving. The problem here that only one additional file is created (and what if it reaches 2Gb as well?). In adition, not all records for old dates might be moved. (I remember a rather funny reply of a helpdesk for such issue: “For the CATEG.ENTRY record 135920009041014.000001, the value date is 18 MAR 3005. As this is a forward categ.entry generated for the PL category 52-344, this is not archived”.) Again, you’ll need to create new enquiries for $arc file. (And if you need a report that takes data from both files you’ll need to develop a nofile enquiry.) Last but not least – if you upgrade your T24 and the file structure changes, $arc file will still keep the old structure – at least it did last time I’ve checked.
• My favourite option – to “distribute” the data file. “Distribution” means that several physical data files can be logically one file for jBASE (and therefore for T24). The procedure is available at the helpdesk on request (in my opinion it’s not complete but “it works”). Of course you can also try to apply “archival” described above and then distribute the $arc file.
Let’s proceed with example of file distribution right now.
Firstly we need to choose a file. It should be not very little file; it should be T24 file (since we’re going to “play” with VOC entries a little to achieve one nice thing that goes beyond the official procedure but makes life much easier).
Browsing the subdirectories of bnk.data with files sorted by size, what we see? aa...
better not to meddle with AA stuff... ac...biggest are stmt.entry and categ.entry...
Could be used despite about 10,000 records only but their @ids are not fit to what I’m thinking about – and I think about organizing parts of a distributed files (promptly named
“part files”) on date principle so we have separate part file(s) for particular periods of time – and then we can store them separately to reduce backup time or even put away from our server and get back only when required. Unfortunately in @ids of both stmt.entry and categ.entry only the system time of their creation presents rather than the bank date and
19 2GB ISSUE; FILE DISTRIBUTION
for distribution we can only use @ids.
Going further... de... for delivery there are their own procedures called “end of period”, or at least there was last time I’ve checked.... eb... why F LOCKING is the biggest? Take a look:
Contents of F.LOCKING C:\sa-tafc> jrunT24.cmd
jsh ~ --> LIST ONLY F.LOCKING
@ID...
Definitely most of this temporary stuff can be removed (but ask helpdesk first). Further on goes F ENQUIRY SELECT which holds selections for all users – also temporary stuff:
Contents of F.ENQUIRY.SELECT C:\sa-tafc> jrunT24.cmd
jsh ~ --> LIST ONLY F.ENQUIRY.SELECT
@ID...
19 2GB ISSUE; FILE DISTRIBUTION
Finally a good candidate was found - it’s a history file of FUNDS.TRANSFER application (which I’ve mentioned above already; it normally becomes very big after some time). Record
@id contains the bank day when it was created and the record goes to the history immediately after COB in most cases (we wouldn’t consider transactions with future value date that might still stay in live file for some time – I’m not 100% sure about that case). But – generally – we can think that if the record went to history then after some time – say, couple of months – the deal isn’t going to create any new records in the history so the part file(s) might reside outside bnk.data and be even readonly. Let’s try this approach.
@id of FUNDS.TRANSFER$HIS looks like that (though in your case it might be different, see AUTO.ID.START>FUNDS.TRANSFER and the helptext or the sample in my previous book in this case):
@IDs of FBNK.FUNDS.TRANSFER$HIS C:\sa-tafc> jrunT24.cmd
jsh ~ --> LIST ONLY FBNK.FUNDS.TRANSFER$HIS
@ID...
19 2GB ISSUE; FILE DISTRIBUTION
FT101861ZC2M;1 FT102070BQB9;1 FT10210G0XJT;1 FT102143R6QS;1 FT10186M9Q0S;1 ...
Here two digits after “FT” represent year number and they are followed by day number in the year. Sorted output tells us that the earliest date for such record is 10186 which stands for.. for...
EVAL helps us again C:\sa-tafc> jrunT24.cmd
jsh ~ --> LIST . SAMPLE 1 EVAL "OCONV(ICONV(10186,’DJ’),’D’)"
DICT ... OCONV(ICONV(10186,"DJ"),"D")
%HOME% 05 JUL 2010
1 Records Listed
...for 5th of July, 2010. Our current bank date is:
DATES listing
jsh ~ --> LIST F.DATES TODAY
@ID... TODAY...
GB0010003 20100809
SG0010001 20100809
GB0010001-COB 20100809
EU0010001 20100809
GB0010003-COB 20100809
GB0010001 20100809
EU0010001-COB 20100809
GB0010002 20100809
GB0010002-COB 20100809
GB0010004 20100809
GB0010004-COB 20100809 SG0010001-COB 20100809
12 Records Listed
19 2GB ISSUE; FILE DISTRIBUTION
Well, that doesn’t give us great time span but we’ll try anyway. We’re putting all records for July 2010 into one file and records that are after that – to another. Firstly we need to create the stub file:
Stub file creation C:\sa-tafc> jrunT24.cmd
jsh ~ --> CREATE-FILE FT-HIS-ALL TYPE=DISTRIB [ 417 ] File FT-HIS-ALL]D created , type = J4 [ 417 ] File FT-HIS-ALL created , type = DISTRIB jsh ~ -->
Now check its current state which was set by default:
Stub file verification
jsh ~ --> VERIFY-DISTRIB FT-HIS-ALL
Partitioning Algorithm is SYSTEM, with delimiter ’-’
There are no Part files defined
Of course we need to change partitioning algorithm which is used by jBASE to deter-mine in which part file particular record is to be found (or placed into). As it was already mentioned, only @id is available for such algorithm. To achieve that we write a subroutine:
Creation of algorithm C:\sa-tafc> jrunT24.cmd
jsh ~ --> JED . FT-PART-ALGO
File . , Record ’FT-PART-ALGO’ Insert 19:52:04
Command->
0001 * algorithm for FT$HIS
0002 SUBROUTINE FT-PART-ALGO(sReserved, sKey, sPartNumber) 0003
0004 nDate = sKey[3,5]
0005
0006 IF ISDIGIT(nDate) THEN 0007
0008 IF nDate LT 10213 THEN
0009 sPartNumber = 1
0010 END ELSE
0011 sPartNumber = 2
0012 END
0013
19 2GB ISSUE; FILE DISTRIBUTION
0014 END ELSE ;* we are cautious 0015 sPartNumber = 99
0016 END
---Then we compile it:
Compilation of algorithm
C:\sa-tafc> jrunT24.cmd
jsh ~ --> BASIC . FT-PART-ALGO FT-PART-ALGO
BASIC_12.c
Source file FT-PART-ALGO compiled successfully jsh ~ --> CATALOG . FT-PART-ALGO
FT-PART-ALGO
Object FT-PART-ALGO cataloged successfully Library C:\sa-tafc\lib\lib0.dll rebuild okay jsh ~ -->
Stop, stop... we are no more in the situation when we can separate a subroutine from T24. To ensure that T24 will work normally with that file we need to put our algorithm to T24 libraries:
Recompilation of algorithm jsh ~ --> DECATALOG . FT-PART-ALGO
Object FT-PART-ALGO decataloged successfully Library C:\sa-tafc\lib\lib0.dll rebuild okay jsh ~ --> set JBCDEV_LIB=<path_to_bnk.run>\lib jsh ~ --> CATALOG . FT-PART-ALGO
FT-PART-ALGO
Object FT-PART-ALGO cataloged successfully
Library <path_to_bnk.run>\lib\lib2.dll rebuild okay jsh ~ --> jshow -c FT-PART-ALGO
Subroutine: <path_to_bnk.run>\lib\lib2.dll
jBC FT-PART-ALGO version 201014.0 Mon Jul 18 19:57:12 2011 jBC FT-PART-ALGO source file .
19 2GB ISSUE; FILE DISTRIBUTION
Now we can add our algorithm to our stub file. It’s a good idea to verify the file after each step:
Attachment of algorithm to stub file
C:\sa-tafc> jrunT24.cmd
jsh ~ --> CREATE-DISTRIB -pUSER,FT-PART-ALGO FT-HIS-ALL jsh ~ --> VERIFY-DISTRIB FT-HIS-ALL
Partitioning Algorithm is USER Subroutine ’FT-PART-ALGO’
User subroutine OK.
There are no Part files defined
Good, exactly where we expected it to be. Create and add part files (note that we need part file 99 to handle exceptions – according to the algorithm; such handling is absolutely mandatory):
Creation and assigment of the part files jsh ~ --> CREATE-FILE FT-HIS-PART1 TYPE=J4 1 101
[ 417 ] File FT-HIS-PART1]D created , type = J4 [ 417 ] File FT-HIS-PART1 created , type = J4 jsh ~ --> CREATE-FILE FT-HIS-PART2 TYPE=J4 1 101 [ 417 ] File FT-HIS-PART2]D created , type = J4 [ 417 ] File FT-HIS-PART2 created , type = J4 jsh ~ --> CREATE-FILE FT-HIS-PART99 TYPE=J4 1 101 [ 417 ] File FT-HIS-PART99]D created , type = J4 [ 417 ] File FT-HIS-PART99 created , type = J4
jsh ~ --> CREATE-DISTRIB -a FT-HIS-ALL 1 FT-HIS-PART1 Part file ’FT-HIS-PART1’, Part number 1 added
jsh ~ --> CREATE-DISTRIB -a FT-HIS-ALL 2 FT-HIS-PART2 Part file ’FT-HIS-PART2’, Part number 2 added
jsh ~ --> CREATE-DISTRIB -a FT-HIS-ALL 99 FT-HIS-PART99 Part file ’FT-HIS-PART99’, Part number 99 added
jsh ~ --> VERIFY-DISTRIB FT-HIS-ALL
Partitioning Algorithm is USER Subroutine ’FT-PART-ALGO’
User subroutine OK.
Part file ’FT-HIS-PART1’, part number 1 - OK Part file ’FT-HIS-PART2’, part number 2 - OK
19 2GB ISSUE; FILE DISTRIBUTION
Part file ’FT-HIS-PART99’, part number 99 - OK
Congratulations, we are ready to copy the data:
Copying of data
jsh ~ --> COPY FROM FBNK.FUNDS.TRANSFER$HIS TO FT-HIS-ALL ALL 682 records copied
Checking how all was copied:
Check of data
jsh ~ --> LIST FT-HIS-PART1 ONLY FT-HIS-PART1..
FT10210K6805;1 FT101973MNRH;1 FT101876Q21B;1 FT10194426BX;1 FT10194HGYRH;1 FT1019435RQ2;1 FT10186HWG3J;1 FT1018658RY6;1 FT10194BHDN2;1 FT10186XMK5T;1 FT101869CJKW;1 ...
jsh ~ --> LIST FT-HIS-PART2 ONLY FT-HIS-PART2..
FT10217WBR9V;2 FT10217CVD22;1 FT102163R16Y;1 FT102172R2FT;1 FT102170KCRS;1 FT102162WT69;1 FT10217M6B8V;1 FT10217ZJPFL;1 FT1021883BK2;1 FT102174B729;1 FT10217K6MNZ;1
19 2GB ISSUE; FILE DISTRIBUTION
...
jsh ~ --> COUNT FT-HIS-PART1 465 Records counted
jsh ~ --> COUNT FT-HIS-PART2 216 Records counted
Well... doesn’t look like we have all 682 records here. That’s why our file 99 was necessary:
Check of data - continued
jsh ~ --> LIST FT-HIS-PART99 ONLY FT-HIS-PART99.
FTAA10186QY2ZR;1 1 Records Listed
We see that the @id doesn’t follow the standard so it went as an exception. The total number of records is correct:
Check of data - finished
jsh ~ --> COUNT FT-HIS-ALL 682 Records counted
We also see here that we can address the individual part files as well as a whole distributed file.
But the file isn’t yet been recognized by T24. To achieve that firstly let’s think: do we need all new data files in the bnk.run? Of course no. We need to move them to bnk.data and then adjust VOC entries for T24 to find all of them. We’ll move FT-HIS-ALL, FT-HIS-PART1, FT-HIS-PART2 and FT-HIS-PART99 to ../bnk.data/ft and then:
Necessary VOC entries - stub file
jsh ~ --> COPY FROM VOC FBNK.FUNDS.TRANSFER$HIS,FBNK.FUNDS.TRANSFER-SAVE 1 records copied
19 2GB ISSUE; FILE DISTRIBUTION
jsh ~ --> JED VOC FBNK.FUNDS.TRANSFER$HIS
File VOC , Record ’FBNK.FUNDS.TRANSFER$HIS’ Insert 20:41:33 Command->
0001 F
0002 ../bnk.data/ft/FT-HIS-ALL 0003 ../bnk.dict/F_FUNDS_TRANSFER]D
End Of Record
---Necessary VOC entries - part file 1 jsh ~ --> COPY FROM VOC FBNK.FUNDS.TRANSFER$HIS,FT-HIS-PART1 1 records copied
jsh ~ --> JED VOC FT-HIS-PART1
File VOC , Record ’FT-HIS-PART1’ Insert 20:45:13
Command->
0001 F
0002 ../bnk.data/ft/FT-HIS-PART1 0003 ../bnk.dict/F_FUNDS_TRANSFER]D
End Of Record
---Do the same with part files #2 and #99 and check it:
Necessary VOC entries - final check jsh ~ --> VERIFY-DISTRIB FBNK.FUNDS.TRANSFER$HIS
Partitioning Algorithm is USER Subroutine ’FT-PART-ALGO’
User subroutine OK.
Part file ’FT-HIS-PART1’, part number 1 - OK Part file ’FT-HIS-PART2’, part number 2 - OK Part file ’FT-HIS-PART99’, part number 99 - OK
As you might have noticed, these VOC entries use the same dictionary – the one for initial history file, so we don’t need the files FT-HIS-ALL]D, FT-HIS-PART1]D, FT-HIS-PART2]D and FT-HIS-PART99]D in bnk.run and you can delete them. Alternatively, we could have used
“DATA” keyword in part files creation command, like:
Creation of data part only for a part file jsh ~ --> CREATE-FILE DATA FT-HIS-PART1 TYPE=J4 1 101
19 2GB ISSUE; FILE DISTRIBUTION
Also we can back up and remove the old data file from ../bnk.data/ft.
Now the final check – log in to T24 and type FT L ; L at “AWAITING APPLICATION”
prompt, then press F5:
T24 now uses a distributed file
Model Bank FUNDS.TRANSFER$HIS - DEFAULT HISTORY LIST
ID RECORD.STATUS INPUTTER FUNCT.
---1 FT10186B51RP;1 MAT 4145_SEAT.USER__OFS_SEAT
2 FT10186BB96G;1 MAT 4899_SEAT.USER__OFS_SEAT
3 FT10186BC4NJ;1 MAT 4145_SEAT.USER__OFS_SEAT
4 FT10186BJ634;1 MAT 291_SEAT.USER__OFS_SEAT
5 FT10186BJNYP;1 MAT 5436_SEAT.USER__OFS_SEAT
6 FT10186BSXJQ;1 MAT 4116_SEAT.USER__OFS_SEAT
7 FT10186BT3NR;1 MAT 7038_SEAT.USER__OFS_SEAT
8 FT10186C5QG3;1 MAT 4957_SEAT.USER__OFS_SEAT
9 FT10186CBZZ5;1 MAT 4360_SEAT.USER__OFS_SEAT
10 FT10186CFWVV;1 MAT 4145_SEAT.USER__OFS_SEAT
11 FT10186CH4PQ;1 MAT 7038_SEAT.USER__OFS_SEAT
12 FT10186CH48K;1 MAT 522_SUJA1__OFS_BROWSERTC
13 FT10186CJWQ0;1 MAT 3789_SEAT.USER__OFS_SEAT
14 FT10186CTSLV;1 MAT 3436_SEAT.USER__OFS_SEAT
15 FT10186D9F66;1 MAT 7285_SEAT.USER__OFS_SEAT
16 FT10186DB93N;1 MAT 7392_SEAT.USER__OFS_SEAT
---18 JUL 2011 21:01:46 USER (09 AUG) VLADIMIR.K [5377,INPAGE 1 >>>3>>>
ACTION
AWAITING PAGE INSTRUCTIONS
If you are still not sure, don’t leave this screen and open another session, then find out the port number for T24 session above and see opened files for it:
Yet another final check
jsh ~ --> WHERE
Port Device Account PID Command
13 ntcon t24 5584 <path_to_TAFC>\bin\jsh -para K.LOG
EX
*14 ntcon Vladimir.Kaz 4608 jsh
WHERE jsh ~ --> LIST-OPEN-FILES 13 (V
19 2GB ISSUE; FILE DISTRIBUTION
Port File name ...
13 ..\bnk.data\ft\FT-HIS-PART99 13 ..\bnk.data\ft\FT-HIS-PART2 13 ..\bnk.data\ft\FT-HIS-PART1 13 ..\bnk.data\ft\FT-HIS-ALL
13 ..\bnk.data\ft\FBNK_FUNDS_TRANSFER#NAU 13 ..\bnk.data\ft\FBNK_FUNDS_TRANSFER ...
You see that while live and $nau files are the same as they were the $his file is repre-sented by our distributed one.
By default, all part files are opened whenever a stub is opened; this can be changed by setting the environment variable jedi distrib defopen to 1.
Now – to decrease backup time – we can think of moving “older” part of the distributed file away from bnk.data. You can try it yourself; the only thing that is necessary apart from moving the file is to change its VOC entry. I even made this file readonly and all still worked.
It’s a good idea also to resize each part file.
Having done that, why not to try another option - temporarily remove that part file pretending there’s no space on the disk. Can we do it? The day numbers in that file were up to (but not including) 10213 so move this file away (or simply rename) and firstly double-check with verify-distrib – it shouldn’t see part 1 (“cannot open – error 2”). Then try to see the implications:
One part file is missing
C:\sa-tafc> jrunT24.cmd
jsh ~ --> VERIFY-DISTRIB FBNK.FUNDS.TRANSFER$HIS
Partitioning Algorithm is USER Subroutine ’FT-PART-ALGO’
User subroutine OK.
Part file ’FT-HIS-PART1’, part number 1 - Cannot open (error 2) Part file ’FT-HIS-PART2’, part number 2 - OK
Part file ’FT-HIS-PART99’, part number 99 - OK jsh ~ --> SELECT FBNK.FUNDS.TRANSFER$HIS
217 Records selected
jBASE now sees less records than it did before. What T24 says? Use FT L ; L command again:
19 2GB ISSUE; FILE DISTRIBUTION
T24 now uses a distributed file with one part missing
Model Bank FUNDS.TRANSFER$HIS - DEFAULT HISTORY LIST
ID RECORD.STATUS INPUTTER FUNCT.
---1 FT10214HKVSZ;1 MAT 514_INPUTTER
2 FT10214J13N8;1 MAT 514_INPUTTER
3 FT10214J60FB;1 MAT 777_INPUTTER
4 FT10214JRRSS;1 MAT 514_INPUTTER
5 FT10214K5FD9;1 MAT 772_INPUTTER
6 FT10214L1TJP;1 MAT 777_INPUTTER
7 FT10214NT8QV;1 MAT 778_INPUTTER__OFS_SWIFTIN
8 FT10214P581G;1 MAT 512_INPUTTER
9 FT10214PSD8J;1 MAT 513_INPUTTER
10 FT10214PYSQN;1 MAT 512_INPUTTER
11 FT10214RC54G;1 MAT 778_INPUTTER__OFS_SWIFTIN
12 FT10214W9RR0;1 MAT 512_INPUTTER
13 FT10214W860N;1 MAT 512_INPUTTER
14 FT10214YS176;1 MAT 514_INPUTTER
15 FT10214YTPFX;1 MAT 299_INPUTTER
16 FT10216CPVP3;1 MAT 762_INPUTTER
---22 JUL 2011 10:57:58 USER (09 AUG) VLADIMIR.K [6059,INPAGE 1 >>>3>>>
ACTION
AWAITING PAGE INSTRUCTIONS
See that the list starts from other @ids than it did when all part files were available?
The behaviour is a bit unexpected to me since I thought that an error message would be raised and we’ll have to set jedi distrib defopen to 1 and correct user enquiries. Anyway, it will serve us well if correction of enquiry is described anyway – otherwise users might end up with wrong reports that miss some data. As a base we’ll use a sample enquiry-related build.routine named ENQ.NARROW from my previous book and rename it to ENQ.WARNING.
ENQ.WARNING subroutine - the beginning
1 *---*
2 SUBROUTINE ENQ.WARNING(P.ENQ)
3 * V.Kazimirchik (KZM), 2011.
4 * Build routine for enquiry to see if all data is
5 * available for an enquiry since particular part file
6 * (1 in our case) might be absent.
7 *---*
8 $INSERT T24.BP I_COMMON
9 $INSERT T24.BP I_EQUATE
10 $INSERT T24.BP I_ENQUIRY.COMMON
19 2GB ISSUE; FILE DISTRIBUTION
Nothing unusual at the beginning. (Note that here “$INSERT T24.BP I COMMON” etc is used instead of “$INSERT I COMMON” to avoid additional parameters to basic command – this wouldn’t cause any harm unless Temenos changes the product name and tries to wipe out all references to the old one as it was attempted with “Globus” before.) Now we open FT history file and get the part files information using statement status. Note enq.error variable that is used to pass an error back to user:
ENQ.WARNING subroutine - continued
11
12 F.FT.HIST = ’’
13 CALL OPF(’F.FUNDS.TRANSFER$HIS’, F.FT.HIST)
14
15 STATUS V.INFO.L FROM F.FT.HIST ELSE
16 ENQ.ERROR = ’FT HISTORY STATUS COULD NOT BE READ’
17 RETURN
18 END
19
20 IF V.INFO.L<21> NE ’DISTRIB’ THEN RETURN
21
22 V.PARTS.L = V.INFO.L<24> ;* part numbers list, @VM-delimited
23 V.FILES.L = V.INFO.L<25> ;* part file names list
We need the file name for the part #1:
ENQ.WARNING subroutine - continued
24
25 * Not sure if "1" is always first, especially if that part was
26 * removed and then added back, so find it.
27
28 FIND 1 IN V.PARTS.L SETTING V.DUMMY, V.POSN ELSE
29 ENQ.ERROR = ’PROBLEMS WITH DISTRIBUTED FILE STRUCTURE’
30 RETURN
31 END
32
33 V.PART.ONE = V.FILES.L<1,V.POSN>
Here we set the threshold for the dates and get the selection from user input:
Here we set the threshold for the dates and get the selection from user input: