FC-TOOLBOX TUTORIAL

Prepared by Bob Muchamore for the FreeCEN Project: (rev. Sept. 2002 )
Discussion {23} of Sept 2002 Changes

{01}

'Minimal skills' {02}
Methods {07}
Data File Editing {11}

FC-TOOLBOX provides access to the FCTOOLS Diagnostic and Administration Routines.

This page is marked to allow following "html links" on printout pages.
The method uses matching {number} at a link and at the "anchor point".
"Anchor points" are sequentially numbered at the left margin.

Each unit {02} has been constructed on a minimal "prompt and reply" basis to assist those who are unfamiliar with ‘command-line’ procedures, or for those who will use FCTOOLS only occasionally.

All routines are resident in the \FCTOOLS\ folder, and working files are established in sub-folders \FCTOOLS\FCTWKG\ and \FCTOOLS\TMP\.

You were asked to install FCTOOLS Software into this folder.
You may find it useful to create a Desktop "Shortcut" to the FCT.BAT programme, and to adjust the Icon Properties for "Run Maximised" and "Close on Exit". An alternative Icon is provided if you wish to change.

In general, original FreeCEN files may be renamed but are not overwritten.


THE TUTORIAL.

This tutorial is designed to be worked in conjunction with files contained in a self-extracting zip file "DEMO9999.EXE "

Copy DEMO9999.EXE to a floppy disk (preserving the original safely so that it may be available for future regeneration if required).

RUN DEMO9999.EXE to expand four files which have been assembled to provide testable examples for the types of files that may be received from FreeCEN Volunteers:-

  1. TRAN9999.ZIP represents a Transcription file prepared using INCENS.
  2. SPRD9999.ZIP represents a Transcription prepared using SSCENS (the Spreadsheet method)
  3. VALD9999.ZIP represents a Validated file prepared using VALCENS, having earlier been Checked.
  4. DEVPARMS.DAT, a standard FreeCEN County File.
DO NOT UNZIP THESE FILES ANY FURTHER
NOR ATTEMPT TO LOAD THEM TO FREECEN ROUTINES

FILENAMES:

FreeCEN and the TOOLBOX use "short-form" filenames.
This means that a file NAME may consist of up to eight characters, and a FILETYPE of up to three more characters, illustrated in the filenames above.

Extended filenames (although permitted by Windows, etc) should not be used.


THE NEED FOR FCTOOLS

It is found that, for a variety of reasons, FreeCEN files can contain flaws or errors which are capable of interfering with smooth operation of later FreeCEN procedures.

The FCTOOLS routines were developed to facilitate detection and removal of such flaws and errors, maintaining a degree of file integrity that cannot be guaranteed if a normal "Text-Editor" is use to make alterations.
Often the flaws detected are, in fact, the result of such "unauthorised" editing!

Some basic operations using FC-TOOLBOX do not require "special skills".
It may be that errors in the files are found which will require delegation to a person with more skills (or perhaps simply more time) to effect the necessary corrections. The basic operations in themselves take only minutes to establish the integrity of a given file.


USING TOOLBOX

While full Technical Instructions are available for those interested, the TOOLBOX has been constructed to operate on a minimal "prompt and reply" basis.

Many 'replies' will be single-key. Any prompt showing a large cursor will need to be concluded with the <return> key.

{02}
Introduction {01}
Methods {07}
Data File Editing {11}

THE MENU

TOOLBOX opens with a Menu, permitting the following selections:

1 QCHK {03} Basic File Examination
2 CSV2DAT {04} Conversion: Spreadsheet to FreeCEN
3 DAT2CSV {17} FreeCEN to Spreadsheet Diagnostic      advanced
4 FIXPARMS {19} For Salvage of Work in Progress      advanced
5 ZIPIT {05} Rebuild zip-files for FreeCEN use
6 VALCONV{22} Update Validated file to latest format
7 TUTORIAL Toolbox Tutorial
8 HOWTO Memory Jogger and Detailed Specifications
9 CLEAN {06} Inspect or Empty Working Folder
0 EXIT Exit TOOLBOX: D to change Drive Letters

"MINIMAL SKILLS" ROUTINES

{03}

1. QCHK

The "Minimal Skills Demonstrations" use the sample file "TRAN9999.ZIP" and does not discuss the results of running any other zip files from the floppy!

This option conducts a rapid check of any FreeCEN file (either Transcribed or Checked) and prepares reports to indicate whether errors may be present which might interfere with subsequent FreeCEN processes.

Select 1 from the menu, read and respond according to the various messages displayed.
Toolbox will copy the TRAN9999.ZIP file before testing and will produce two error summaries. Note that errors are reported for two tests which have been conducted.
The next screen "IF ANY ERRORS..." is informative, but asks "Do you wish to examine the error files now?".
Please respond with a "Y".

We read that there are four errors involving "re-aligning" and two errors involving "Carriage return detected".
**** These are all file defects which could cause problems during Checking.
Hit any key to continue: the second error report is presented.
Two errors are reported where (apparently) the registration district has changed during data entry.
**** This is an error unlikely to be found and has been engineered to create this demonstration.
Hit any key to continue, eventually returning to the Menu .

Output:
The two error files (in this example named RG129999.ERC and RG129999.ERD) are added to the (Transcriber's) floppy disk as evidence that the test has been run.
QCHK has used the folder \FCTOOLS\TMP\ to hold temporary files which it deletes on completion of the task, thus avoiding "computer clutter".

Interpretation:
In practice most files will pass QCHK with no errors reported.
This would indicate that, while the data may not be perfect, the file is unlikely to cause future processing problems.

In this particular example the errors reported require that the file should be examined in detail and rebuilt - a job that will require additional expertise and may be delegated.
The error reports may be opened or printed outside TOOLBOX, using Notepad.


{04}

2. CSV2DAT

A common requirement will be the conversion of SSCENS Transcribed Spreadsheet files into a format that can be Checked using FreeCEN.
See also CSV2DAT {17} for Editing Applications.

This demonstration uses the sample file SPRD9999.ZIP to simulate such a Spreadsheet Transcription.

Select 2 from the menu.
We are only running a "no-skills" operation so "space" through the "Option Switch" selections.
"You have selected defaults" ...

Enter Drive Letter - note that this routine can operate on a file which is present on floppy OR which has been loaded into the Working Folder. The file may be in zipped or plain form.
We shall enter N
There may be multiple files on the floppy (as in this instance): Select SPRD9999.ZIP

The routine will expand the file into the working folder: the next listing has only one .CSV file present:
enter RG129999.CSV
The routine runs and produces an Error Summary "0 errors, 30 warnings"
"... examine the file now?" - Y

Again, realise that this particular file was engineered for demonstration purposes and that normally such warnings would not be appropriate!
Also, note that these are WARNINGS (as opposed to "errors") and as such are capable of being corrected as necessary during subsequent FreeCEN processes.

Hit any key to continue
Note that the Error report is retained in the \FCTOOLS\FCTWKG\ folder
Continue to the Menu {01}

For confirmation purposes - Select 9 - DO NOT DELETE!

Interpretation:
The Spreadsheet file RG129999.CSV has been successfully converted to a FreeCEN file RG129999.DAT ready for CHEKCENS.

Any key except D in order to return to the Menu {01}

{05} It remains to create a ZIP file of the type required for CHEKCENS input.

5. ZIPIT

At this level, TOOLBOX will guide the creation of an appropriate ZIP file.
Select 5 from the Menu.

The second page recalls that there are different needs for the various FreeCEN zip files. We need to prepare a TRANSCRIPTION file.
Select 1
Note the requirements for CHEKCENS input: a DATA file, a ctyPARMS file and "MYNOTES.TXT"
(Be warned that not all Transcribers produce MYNOTES.TXT. This is to be regretted, but the presence of such a file is not vital to subsequent Checking.)
Continue...

NOTE the files that are listed as available: there is no ctyPARMS.DAT file.

In this instance we are dealing with "a Devon Piece" and need to copy DEVPARMS.DAT into the \FCTOOLS\FCTWKG\ folder. By strange coincidence there is a copy on the floppy disk!

You may be able to do the copying by putting TOOLBOX into background for a while...
If necessary restart ZIPIT and get back to the files list.
You are asked to name the three files to be included in the zip.

The ZIP file will be created onto floppy disk.
This can be a Blank disk, or the file will normally be added to the disk on which the CSV file was received.
Answer the prompts accordingly.
Enter / since there is no pre-existing file which is named in a manner which can cause confusion.
You must now provide a name for the new file. This is comprised of your own FreeCEN code + the piece number.

One last check: the screen summarises your proposals.
If you agree, the zip file is created on the floppy disk, and you are returned to the Menu {01}

KEEP A SECURITY COPY of this file before sending the work off to a Checker!

{06} Get into the habit, as each "project" is completed, to Clean out the Working Folder.

Select 9: Select D

You are returned to the menu.


CONGRATULATIONS

You have mastered the essential "minimal skills" expected for all Co-Ordinators.

Comments and queries are always welcomed via the Co-ORDS Mailing List.


{07}

METHODS

Introduction {01}
Minimal skills {02}
Data File Editing {11}

Within the requirements that FreeCEN files for loading to the Database shall be accurate and defect free, it is possible that each worker will derive their own operating method.

It is vital that persons attempting to use Advanced Techniques should be careful and accurate workers, with a good understanding of the FreeCEN ethos and an understanding of National Census returns.

Error Reports {08}
File Preservation {09}
Associated Files {10}

Using the FCTOOLS routines (either with TOOLBOX, or using "command-line" techniques) operators will be expected to

  1. receive copies of FreeCEN files, preferrably accompanied by the films
  2. examine the content and structure of critical Data Files
  3. make necessary corrections to those files
  4. re-examine and re-correct as necessary
  5. re-assemble the FreeCEN file sets to suit following processes

These functions are discussed in detail under Advanced Techniques {11}


{08}

ERROR REPORTS

[ Back to Methods ]
Apart from certain FATAL errors (which will cause the routines to stop operating, but which will normally be reported on-screen), FCTOOLS produces a File "Error Report" containing degrees of information:
  1. "ERROR" advises of a defect which MUST be corrected before testing is re-run to ensure that the defect is cleared.
    Defects in this category are liable to cause malfunction in subsequent stages of FreeCEN processing, thus involving considerable lost time and wasted effort!
  2. "WARNING" advises of a deficiency which NEEDS correction.
    Such items will not normally be fatal to subsequent processing but will detract from the integrity or accuracy of the end product DataBase
    Frequently this type of error is easier corrected while the data is in spreadsheet than correction using CHEKCENS or VALCENS.
  3. "INFO" advises of an apparent discrepancy between an item and its neighbours (e.g. Page Numbering should be sequential within an Enumeration).
    INFO reports do not necessarily imply a data defect, but draw attention to the fact that potential for an error exists.
    Such items will not affect FreeCEN processing but could result in inaccuracies for the DataBase.
Error Reports produced from CSV2DAT are named with a "filetype" .ERC; from DAT2CSV are type .ERD
Successive reports will over-write any previous error reports present in the Working Folder!
{09}

FILE PRESERVATION

[ Back to Methods ]

It is preferred that FCTOOLS Output Files are generated onto the same floppy disk as the incoming data.
This will enable the "history" of a Census Piece to be available if needed for future examination.

To this end TOOLBOX adopts a policy of renaming files that become replaced by rebuilding:

Note that both Checked and VALCENS-Validated Data files carry a filetype .CHE
The specific state of a .CHE file must be determined by the nature of associated files contained in the ZIP package.


{10}

ASSOCIATED FILES

The contents of ZIP files, "output" by FreeCEN routines for use in "the next processing", MUST contain at least the following file sorts:-
Note that "RG" will be replaced by "HO" for the 1841 and 1851 census, and by "RS" or "HS" for Scottish Census files.
Transcription
Checking
Validation
Description
RGnnpppp.DAT
RGnnpppp.DAT
*RGnnpppp.DAT*
Transcriber's Data
ctyPARMS.DAT
ctyPARMS.DAT
ctyPARMS.DAT
County Details
MYNOTES.TXT
RGnnpppp.TXT
*RGnnpppp.TXT*
Trans./Check Notes
.
RGnnpppp.CHE
RGnnpppp.VLD
Checked or Validated Data
.
RUNPARM.DAT
*RUNPARM.DAT*
routine file
.
.
*PLACECHG.DAT*
Alternative Birthplaces
.
.
*UKCENSIN.DAT*
County Identifier
.
.
*BACK_UP.BAT*
routine file
.
.
*IMPORTIT.BAT*
routine file
.
.
VALNOTES.TXT
Vald. Notes - if written

Files from Validation marked with *...* are not required in VALDREV ZIP output, but are included in VALCENS output.

[ Back to Methods ]


ADVANCED TECHNIQUES

{11}
Introduction {01}
Minimal skills {02}
Methods {07}

Recapitulation:

receive file {12}
examine {13}
DAT2CSV (editing) {14}
INCENS to SSCENS {15}
CSV2DAT {17}
Correct and Re-check {18}
FIXPARMS {19}
Zip files {20}
CSV types {21}
VALCONV {22}
.
.

This demonstration uses FCTOOLS routines, with TOOLBOX, to:

ALWAYS make sure that you have a spare copy of original files (thus protecting against loss if things don't work out) before starting any Data File Editing! Your emergency copy may be in a non-FCTOOLS folder, or on floppy, to suit your own preferences.

Because of the nature of the task, it is recommended that
Data File Editing should be undertaken in an environment having minimal distractions!



{12}
Any indented text, set using this colour, is concerned with matters not relevant to the first demonstration using the TRAN9999.ZIP file.
Such text will have greater relevance to later examination of VALD9999.ZIP and other work, and study may be deferred.

"1. Receive copies of FreeCEN files"

[ Recapitulate ] {11}

FCTOOLS is concerned with the structure and content of CENSUS DATA files, whether these are
"as Transcribed", "Checked", or "Validated".

Before commencing any new work, check that the working folder has been "Cleaned" (Menu #9).

Files may be accepted from floppy disk, or (e.g. via email) loaded direct into the \FCTOOLS\FCTWKG\ folder.
The data may be in "text" form, or may be "zipped". TOOLBOX will handle any necessary "unzipping"

Discussion will concentrate on the assumption that incoming files are available on floppy disk.


{13}

"2. examine the content and structure of critical Data Files"

[ Recapitulate {11}]

It is always worth running QCHK to obtain a quick evaluation of the possible state of a file.
This operation has been discussed in Minimal Skills {02}

RUN DAT2CSV (Menu #3)
Possible uses for the .CSV file produced need to be discussed individually
Click for a Detailed Schedule of DAT2CSV Errors
any key to continue
{14}

DAT2CSV for File Editing

[ Recapitulate {11}]
The second page permits selection of "Option Switches"
Choice from the options offered in "Group 1" will affect the nature of the .CSV file produced, while those in "Group 2" determine the degree of Error Reporting
Only one Option may be selected from each Group
Group 1
Option a will cause a conversion of every field in the FreeCEN file to a corresponding spreadsheet 'cell'.
This type of display is useful to detect "pattern variations" when rapid-scrolling down the spreadsheet.
The default. (space, or b), produces a skeletal .CSV file without a lot of repetitious detail.
Option w is not relevant to File Editing. See WWW {16}

MAKE TIME, if necessary, to compare {21} and understand the different styles of CSV file that we are considering!

Group 2
"default" (space), produces an "standard report"
q (=Quiet), produces a minimal report which details serious defects
v (=Verbose), produces a comprehensive report of defects, warnings and Information

The routine will not permit invalid selections.
Choose default (space) for each Group.
Note that you may Quit the routine, or amend your Option choice.

The routine next quizzes for the location of the input file.
(If you are not working from floppy, the file must be in the \FCTWKG\ folder associated with the FCTOOLS programmes)

Enter N
Because the floppy disk may contain several files, you are required to name the input file
Select TRAN9999.ZIP
The file is unzipped to the working folder; Enter filename ... - don't forget the extension!

Note that the file-type.VAL, although not an official FreeCEN "type", may be accepted as an input to DAT2CSV
This will only be applicable during "re-examine and re-correct sequences" where the original file has already been FreeCEN Checked or Validated.

Select RG129999.DAT

The compiled DAT2CSV takes over and converts the file before handing control back to TOOLBOX
Observe that the report is for 6 errors, 0 warnings
... examine the error file ... Y
These are errors which will disrupt operation of the FreeCEN CHEKCENS routine!

hit any key...
Note the screen comments; hit any key...

When the Error Report is not too extensive (some can be hundreds of items!) it can usefully be printed out from Notepad for handy reference.

Do not "Clean the Working Folder yet!

We will continue this discussion for this particular file in Section 3. make corrections {17} ...


{15}

To convert INCENS files for completion by SSCENS Data Entry

[ Recapitulate {11}]

The DAT2CSV procedures for examining and correcting should be followed as detailed for the various stages of file editing

The CSV file must be produced using Group 1 "default" option
"Correction" of the file should be attempted where practical, after which outstanding errors can be brought to the attention of the Transcriber.

The file to be returned to the Transcriber will be type .CSV
ctyPARMS.DAT and MYNOTES.TXT will already be in the Transcriber's possession.
ZIPIT is not relevant, although it may be advantageous to compress the file before emailing.


{16}

Internet Display

[ Recapitulate {11}]
This is a complex procedure and should only be attempted by persons having Programming experience.
Full Specifications are available.


{17}

"3. make necessary corrections to those files"

[ Recapitulate {11}]

It is important to realise that FCTOOLS will correct many file defects on its own, during conversion of the CSV file back to FREECEN formats.
For the very little time involved it is also worth obtaining the CSV2DAT Error Report before beginning any attempts to correct the file manually.

RUN CSV2DAT (Menu #2)
Possible uses for CSV2DAT include any key to continue

The second page permits selection of "Option Switches".

The "Group 1" Rules for CSV2DAT Options are simple:

If working with a Checked file, Option c MUST be used
If working with a Validated file, Option d MUST be used
If the CSV file derived from DAT2CSV Option a then CSV2DAT MUST also use Option a
If the CSV file derived from DAT2CSV default then CSV2DAT MUST NOT use Option a
NEVER attempt to rebuild a VALCENS Validated file using Option d!

Ignore these rules and produce GARBAGE!

Group 2 Options determine the degree of Error Reporting, and selection may differ for DAT2CSV and CSV2DAT.

Group 1
Option a will cause a conversion of every field in the .CSV file .
This option must be applied only to a CSV file which was prepared using "Option a" during the DAT2CSV run
Option "space", (or b), as default produces a FreeCEN file from a "skeletal CSV" file.
Option " c " MUST be applied if rebuilding a file which has been Checked
Option " d " MUST be applied if rebuilding a file which has been Validated
Option " c " or " d "can be combined with " a " to produce an appropriate result

MAKE TIME, if necessary, to compare {21} and understand the different styles of CSV file that we are considering!

Group 2
"default", (space), produces an "standard report"
q (=Quiet), produces a minimal report which details serious defects
v (=Verbose), produces a comprehensive report of defects, warnings and Information

The routine will not permit invalid selections.
Because our previous exercise was to convert TRAN9999 (a "Transcriber" file), Select defaults from each Group.

Options c or d, either alone or with a, MUST be used to rebuild a Checked or Validated file. NEVER attempt to rebuild a VALCENS Validated file using Option d!

Note that you may Quit the routine, or amend your Option choice.

The routine quizzes for the location of the input CSV file
Normally (as in this instance) a CSV for "rebuilding" will exist in the \FCTOOLS\FCTWKG\ folder as a result of earlier DAT2CSV operation.
Enter W for the Hard Drive Working Folder

The folder contains various files: you are required to name the input file
Select RG129999.CSV

The compiled CSV2DAT takes over and converts the file before handing control back to TOOLBOX
Observe the Error Report is for 0 errors, 30 warnings
... examine the error file ... Y
Note that our earlier errors have vanished.

Note the nature of the warnings:
"not sequential" may indicate that some numbers are missing from the series, or they may be out of sequence. Can be corrected in CHEKCENS.
"too many lines": CSV2DAT anticipates 31 lines (maximum) per census page. This figure may be exceeded depending on the way the Enumerator entered details. CSV2DAT has no way of checking the accuracy of "Page Number" as entered by the Transcriber. This can be corrected in CHEKCENS if necessary
"first page" & "first schedule" cannot be checked without reference to the fiche. In this instance it is the result of the file being "engineered" for demonstration purposes

Click for a Detailed Schedule of CSV2DAT Errors.

Note that the Error Report also advises values for counts of records and households
It also confirms that the new file is called RG129999.DAT and is retained in the \FCTWKG\ folder.
(Realise that this will have over-written any previous RG129999.DAT existing in the folder, but the original of this file will still be available on the TRAN9999.ZIP file supplied. The same situation applies to any Transcriber's .DAT file.)
When a file is converted using Option c (alone, or with Option a) CSV2DAT assumes that the original data file was type .CHE The new file is named type .VAL
Be alert! Any .CHE file will be rebuilt as a .VAL file!
Never assume that a .VAL file has been Validated by FCTOOLS.
hit any key...
Note the screen comments; hit any key...

Do not "Clean" the Working Folder yet!

HINT: For instances of long Error Reports on screen, you need not continually "hit any key" at --More--;
q and then q again will abort the display and return you to TOOLBOX.

Interpretation:
In this particular example we have now discussed both Error Reports. The final report suggests that there are no file defects remaining which are liable to cause problems during CHEKCENS operation

Nevertheless, we still need to proceed to "Section 4. re-examine ..."


{18}

4. re-examine and re-correct as necessary

[ Recapitulate {11}]
Interpretation:

It should be understood that each Error Report produced by CSV2DAT above relates to the CSV file which is currently in the Working Folder.
DO NOT RUN DAT2CSV again! since this could negate some corrections you have already made.

Correct methods will involve assessing the nature of the errors and making further corrections to the CSV (in Spreadsheet) before again running CSV2DAT so that a new Error Report can be assessed for improvements in the file.

For a badly corrupted data file, this may involve many repeats of "Section 3. make necessary corrections ..."

Do not "Clean" the Working Folder yet!

Interpretation:

RG129999.DAT, held in the Working Folder, can be corrected using FreeCEN routines.
We may therefore proceed to "Section 5. re-assemble" {20}

BUT, first, read on a while:

Experience helps, but Detailed Specifications including listings of the defects detected by FCTOOLS, and suggested corrective action that will be necessary, can also be accessed via TOOLBOX, Menu 8 .
Follow links for the DAT2CSV or CSV2DAT pages as appropriate to read the relevant error message listings.
These are not as complicated as may appear at first sight, review the text found earlier Error Reports {08}


{19}

FIXPARMS

Section 5. re-assemble {20} [ Recapitulate {11}]

FIXPARMS must be run will be following salvage of a Transcription file when data entry is yet to be completed using INCENS.

It is NOT required if data entry is to be completed using SSCENS!

The INCENS (or WINCENS) routines for Data Entry use a "County" file of the type "ctyPARMS.DAT" where "cty" is the appropriate "Chapman" County Code.
Apart from recording the Transcriber's ID and which Census Piece is currently being worked, this file is progressively updated to record the "Number of Records" and the "Last Household Number" used in the file.
If these values get out of alignment with the data file (e.g. RG129999.DAT) then INCENS runs the danger of storing new records in a manner that can over-write pre-existing transcriptions.

RUN FIXPARMS (Menu #4)
Two screens outline the function and method.
any key to continue
A list of available ...PARMS files is displayed
Note: any copy of ctyPARMS file can be used provided it is relevant to the county of the rebuilt Data File

Normally the correct file should have come to you in the Transcriber's .Zip-file.
Enter filename ... DEVPARMS.DAT
Enter PIECE name You will recall we have been correcting RG129999 (NO .EXTension needed!)
Do you need ... numbers... . Enter Y , just in case you forgot to write them down earlier!
The routine displays the RG129999.ERC Error Report, at the end of which are the necessary numbers, 2067 and 532
continue
Change the ID (code)
Enter New Name
Enter Piece No. (9999)

This file has been engineered to match the original RG129999 demo files.
Note that we have (using FCTOOLS ) deleted some records.
Enter the new ... counts
Ignore Civil Parish ...
Enter Y to set
You have only the one Piece to update:
Enter Piece number <RETURN> (to end FIXPARMS)

... Type Y ..., returns to Menu

Click for a Detailed Description of FIXPARMS.


{20}

5. re-assemble the FreeCEN file sets

[ Recapitulate {11}]
Select ZIPIT (Menu #5)
We have earlier discussed the use of ZIPIT among the "Minimal Skills" {05}
Currently the necessary files are all present in the Working Folder - you didn't Clean it out, did you?

We are to re-assemble a "transcriber's" ZIP file for input to the CHEKCENS routine:
Select 1
Check! Is the appropriate floppy disk in position to accept the output?
name the files
Check the "proposal"

When a file is converted using Option c (alone, or with Option a) CSV2DAT assumes that the original data file was type .CHE The new file is named type .VAL
ZIPIT will rename a VAL file to the "expected" .CHE before assembling the ZIP file
The original .CHE AND the re-built .VAL files are retained in the Working Folder.

GO...
TOOLBOX returns you to Menu

Open Desktop Explorer to examine the Floppy Drive.
Satisfy yourself that the original ZIP file has been renamed, and that the new ZIP file is present.

The Transcriber's original zip file will be renamed as type .ZZZ
The Checker's original zip file will be renamed as type .ZZC
A Validator's original zip file, produced in VALCENS, will be renamed as type .ZZV
A Validator's original zip file, produced in VALCENS, but reworked through VALCONV, will be renamed as type .ZZR
A Validator's original zip file, produced in VALDREV, will be renamed as type .ZZX

Note that these renamed files cannot be unzipped again until renamed again under their former type .ZIP
(The renaming of the old files will not normally affect County working of Pieces. Renaming is done solely to preserve the "history" of a Piece.)

Make sure that you keep such Archive copies of files as may be necessary to replace the content of the (working) floppy disk should the worst happen to the disk or the data on it! You may choose to Archive onto a floppy or a separate folder on a hard drive or even CD.

All copies taken? ...
Close Explorer and return to TOOLBOX...
Select "Clean ..." (Menu #9)

Take a Break - you've earned one!
[ Recapitulate {11}]

{21}
COMPARISON between the various .CSV files that can be produced [ Recapitulate {11}]

Ideally this exercise would not be undertaken while a file examination is in progress since it will introduce "unwanted" files into the Working Folder. However, it will not interfere with any tests in progress, and will show the benefits of ensuring that the working folder is cleaned before starting any new task.

You will run repeated tests on a named file observing, using a Spreadsheet programme, the different results for different selections from the Group 1 Options.

Test #1
Select DAT2CSV (Menu #3)
Examine TRAN9999.DAT (TRAN9999.ZIP)
Group 1 - default
We are not interested in the Error Reports. Proceed with DAT2CSV until you get back to the Menu
Use the Spreadsheet to open RG129999.CSV from the folder \FCTOOLS\FCTWKG\
The TUTORIAL text refers to this as a skeletal display
This conforms to the work produced by a Transcriber using "SSCENS".
Note, in particular, the empty spaces in the left-hand columns... data is entered only when detail changes.
Note the Headings (Row 2) for columns A and B ("Civil Parish" and "ED" - for "Enumeration District")
Scroll right, and note that the last used column is Y, Notes

CLOSE Spreadsheet and return to TOOLBOX

Test #2
Select DAT2CSV (Menu #3)
Examine TRAN9999.DAT (TRAN9999.ZIP)
Group 1 - a
We are not interested in the Error Reports. Proceed with DAT2CSV until you get back to the Menu
This test will have overwritten the CSV and error files from Test 1.
Use the Spreadsheet to open RG129999.CSV from the folder \FCTOOLS\FCTWKG\
The TUTORIAL text refers to this as a Full Dump
Note that three Heading rows are not present!
Every piece of data from the file is now displayed. (Column A is used in INCENS mainly to indicate whether a record is flagged to be "Deleted")
The "Civil Parish" is now column E (Devonport); the "Enum. Dist." is column F; and "Notes" (where written) are in column "AB"

It will be realised that CSV2DAT could make completely different output files from these two examples!

If tested, it would be seen that a .CHE file can produce two further variants of CSV layout caused by the inclusion of a "Eccelsiastic Parish" Column, and a .VLD file contains two extra columns for the "alternative Birthplace" information.

Realise that CSV2DAT could make completely different output files from each of the four possible examples!

The Rules for CSV2DAT Options are simple:

If working with a Checked file, Option c MUST be used
If working with a Validated file, Option d MUST be used
If the CSV file derived from DAT2CSV Option a then CSV2DAT MUST also use Option a
If the CSV file derived from DAT2CSV default then CSV2DAT MUST NOT use Option a
NEVER attempt to rebuild a VALCENS Validated file using Option d!

Ignore these rules and produce GARBAGE!

Circumstances, or personal preference, will have determined whether Option a is selected for DAT2CSV.

NOTE that any data file rebuilt using "option a" will copy exactly the information from each cell of the spreadsheet.
It will normally be found preferable to work with default options and to rebuild a data file from a "skeletal" spreadsheet. This will ensure correct household sequence numbering and separation of Schedules.

Browser [Back] to return to Text - or [ Recapitulate {11}]

{22}

6. VALCONV

Uploading to the Internet Database will accept only files which are compatible with the VALDREV Validation output.

The VALCONV routine takes a VALDpppp.ZIP file produced using VALCENS and converts it to the VALDREV format.
Minimal computer skills are necessary to operate VALCONV, but careful attention to the various message windows which appear require appropriate keyboard inputs.

Since the VALD9999.ZIP file included in the DEMO download was produced to VALCENS format it provides a suitable input for practice using VALCONV.

Select 6 from the Menu. The introductory screens allow opportunity to Quit should you wish.

"You will need the Validator's ZIPfile in the Floppy Drive"
NOTE that VALCONV requires use of the MOUSE!
"Type Q to Quit: any other key to continue"

VALCONV reminds you of the need for a floppy disk: Click "Continue"
A window "VALCENS EXPORT PIECE SELECTION" appears. Click to highlight the required Piece, and Click on [Select]
A Progress window requires you to "Click OK to continue": Close the black DOS screen by clicking on the top "X"

A new window appears, either listing any .TXT files present which may contain Notes written by the Validator or advising that there is no suitable file present in the Zipfile
Either make highlight a choice from the file list and "Select"; or acknowledge that there is no file present.

VALCONV proceeds with conversion - watch the record-count progressing in the window
Close the black DOS window ("X")

A message "SERIOUS ERROR!!!! No CHE FILE" is indicative of an invalid disk - and will result if a file already compliant with VALDREV is offerred for conversion!

It may appear that you are back at the opening window - but the messages have changed!
The detail of these messages is purely academic if you are using TOOLBOX.

Close VALCONV by clicking on the [Quit] button.
You will need to click anywhere on the blue window to bring TOOLBOX to the foreground.
Now hit any key ...

If you have received ANY error message during running of VALCONV: Enter Q to Quit.

The converted file should be copied to floppy immediately to preserve the individuality of each Piece.

If you are copying to the original Validator floppy disk the routine will provide a list of any .ZIP files contained on that disk, and will offer a chance to rename the original VALD... file.
This file will be renamed with a file-type .ZZR
You will be asked to confirm the Piece Number being worked (FOUR digits). NOTE that this piece MUST be available in the \FCTOOLS\EXPORT\ folder.

The file will be copied as required

Operation returns to the TOOLBOX Menu.

ALWAYS use Desktop Explorer (or similar) to confirm that the required file has been saved to floppy.
The nature of things means that the VALDREV compliant zip file will be about half (or less) the size of the original VALCENS file.
[ Recapitulate {11}]


{23}

The revisions introduced Sept 2002 are associated with a re-write of the FreeCEN Validation software which produced significant function changes

. Validated using VALCENS Validated using VALDREV
Data File type .CHE .VLD
Data Record Length 276 bytes 299 bytes
. MUST be used with PLACECHG.DAT Records are self-contained
ZIPfile content ten files three files
Spreadsheet edit? limited fully editable
Delete Records? NO YES
Insert Records? NO YES
Move Records? NO YES
Database Upload Not compliant YES
. Produces GARBAGE if CSV2DAT uses Option "d" .
. Use VALCONV to change to the new standards .

The following summary draws particular attention to changes in the Tutorial, or with regard to Advanced operations:-

{02} The Menu now includes Selection 6 = VALCONV - Update Validated file to latest format

{09} Additional "renamed" file-types for ZIPfiles are defined

{10} Table revised to show files required for .VLD files

{17} CSV2DAT: new option " d " introduced for use with Validated files
option " c " now restricted for use only with Checked files
"Group 1 Rules" amended accordingly

{20} New filetypes .ZZR and .ZZX defined to preserve 'historic' files

{21} "Group 1 Rules" revised (as {17}

{22} NEW Section discusses working of VALCONV
[ Return to Start ]