Contents Home Top Prev Next
Map Refs Pics Search Pages

Text Analysis - Transcription of the Text

Introduction

The main mystery of the Voynich MS is clearly its unknown writing. This topic is addressed from three different aspects, on three (sets of) pages:

This page addresses the second part, the transcription of the text. It summarises historical transcription efforts and describes the present state of things. This page is very relevant for those who are interested in the interpretation of the Voynich MS text, but it can be skipped by those who are not.

A section at the bottom of this page explains where one may find the main transcripion files.

Purpose

The term transcription is used here to describe the conversion of the handwritten text of the Voynich MS into a computer-readable format (file). The purpose of this is to allow computer software to analyse the text, for example in order to derive statistics or to aid the interpretation and ideally translation of the text. The first time that this type of transcription was exercised was by William Friedman after WWII, for processing by so-called 'tabulating machines'.

Many years before that, in the 1930's, Fr. Th. Petersen of the Catholic University of America already made a complete hand transcription (i.e. a hand-written copy) of the MS using a complete photocopy of the MS and in some cases, in order to transcribe difficult parts, the actual MS. This document, in which he added many interesting annotations, is still preserved in the William F. Friedman collection of the Marshall Library in Lexington (Va) (1).

A transcription should be clearly distinguished from a proposed translation of the text. The only purpose of a transcription is to represent each handwritten character in the MS by a symbol in a computer-readable file in a consistent manner. It doesn't really matter which symbol is used for which character. Ever since the 1940's, different people have used different conventions, or transcription alphabets.

History

FSG

The earliest well-known transcription of large parts of the Voynich MS was made in the 1940's by the so-called First Study Group (FSG) of William Friedman, which was briefly described here. The group had defined a transcription alphabet agreed by all members of the team. Because of the desire for secrecy by Friedman, for a very long time nobody outside his team was aware of this transcription exercise or the alphabet they used.

The Friedman transcription was never meant to be a complete transcription of all text in the MS. They decided to concentrate on the linear text written in paragraphs, and not bother with the complicated diagrams with labels and text written in circles. This is very clear from handwritten annotatations made on the source copies saying "don't punch this" or "punch only this".

The FSG transcription alphabet uses capital letters and numbers. It has a rather unusual method (by later standards) for transcribing the 'intruding gallows', namely by using a special symbol (Z) for the pedestal (ch).

The FSG transcription is described in detail in Reeds (1995) (2). A printout of this transcription was discovered by Jim Reeds in the above-mentioned Marshall library, and together with Jacques Guy he entered it in computer readable form. The resulting file is >> still available for downloading. This transcription is far from complete for the reasons mentioned above.

Bennett

In 1976 Prof. William R. Bennett of Yale University published a book on the use of the computer to analyse certain properties of language and text (3). In this book, he dedicated a chapter to an analysis of the Voynich MS text, and for this purpose he needed a transcription alphabet. Bennett's alphabet has not been used outside of this publication. The transcription was made from photographs and stored on paper tape. As reported by Brumbaugh (4), Bennett was assisted in his Voynich MS analysis by Jonina Duker, who was then a sophomore student (5).

Currier

Roughly at the same time, the cryptanalist Prescott Currier made a major transcription effort, using an alphabet he designed independently. This alphabet uses the capital letters A-Z and the numbers 0-9 (i.e. all 36). It does not represent some characters which the FSG alphabet does, and it uses single characters for what appear to be composite characters in the MS. When Currier presented his findings at the 1976 Voynich MS symposium, Mary D'Imperio suggested that it would be important that all researchers use a unified alphabet. She had also started transcribing parts of the MS using her own alphabet, which she abandoned in favour of Currier's, and the two files were merged. The combined transcription of Currier and D'Imperio, and the transcription alphabet designed by Currier, have been used well into the 90's, also in the earliest years of the internet.

The following table shows the three above-mentioned alphabets together.

Char Bennett FSG Currier   Char Bennett FSG Currier
D 4 4   I I I
O O O   ILIE G
S8 8   IILIIE H
GG 9   IIILIIIE 1
Z2 2   IQIR T
LE E   IIQIIR U
QR R   IIIQIIIR 0
CTT S   UL D
ETS Z   NN(*) N
HH P   M M(*) M
PP B   IMIIIL 3
KD F    K J
FF V    IK K
CHTHZ Q    IIK L
CPTPZ W    IIIK 5
CKTDZ X    (6) 6
CFTFZ Y    (7) 7
A A A   Y Y (n)
C C C   V V (v)

(*) Note: Tiltman used the FSG alphabet, but instead of N and M wrote IL and IIL.

One thing that immediately appears from this table is that the various transcribers did not agree on what constitutes a single character in the Voynich MS text, and this turns out to be one of the most difficult points for transcription. Later, Jim Reeds showed that many characters in the Voynich MS cannot be represented exactly by any of the existing alphabets. There are some 'rare' characters, and in addition there are what appear to be ligatures of several characters. These characters have come to be called 'weirdoes', though here I will use the term 'rare characters'. Jim Reeds produced a document with an overview of these rare characters, calling them 'X1' to 'X128'. An excerpt of this little known document is shown below.

Transcription in the age of the internet

With the emergence of the internet and the start of the Voynich mailing list, transcription of the parts that were still missing from Currier's and D'Imperio's efforts was continued using Currier's alphabet. A number of characters using lower case were added to this alphabet. These are already included (in parantheses) in the above table. As part of this new transcription, a more or less standard format for transcription files was introduced, about which more will be said further below.

In parellel to this, a completely new type of transcription alphabet was designed:

Frogguy

The 'Frogguy' transcription alphabet was designed by Jacques Guy in 1991. It uses lower-case characters, numbers and diacritical marks. Rather than representing complete characters, it represents the 'strokes' of the script of the Voynich MS. As a result of this approach it manages to produce the closest similarity of the transcribed text with the script. At the same time, some characters in the MS which may be assumed to be a single character are represented by several using this alphabet. By its nature, it is the first alphabet that allows to properly represent many of the ligatured characters that were mentioned above. Jacques Guy also introduced the so-called 'capitalisation rule' which means that a character that is connected to the following one should be represented by a capital letter. The frogguy transcription alphabet and its use are explained in more detail at a preserved copy of >> one of Jim Reeds' web pages.

An additional important contribution from Jacques Guy was a tool that allowed translation of transcription files from one transcrition alphabet to another. This tool was called BITRANS, was written in Pascal and ran in DOS (it still would). It could handle complex translation rules, such as needed for the Currier alphabet, where, for example, the translation of the character i depends on context.

The frogguy alphabet is the closest in appearance to the Voynich MS text, and it solves the main inconsistency of the Currier alphabet, namely that Currier synthesised strings of consecutive i characters into single characters, while this was not done for strings of consecutive e characters (6). However, it has a distinct disadvantage in that it is not very well suited for the type of numerical analysis mentioned at the top of this page, primarily because of its frequent use of the apostrophe, and to a lesser extent the mixture of letters and numbers.

When Gabriel Landini and myself embarked on a new transcription of the MS, we decided that a new alphabet, but one based on the principle to Frogguy, would be needed.

Eva

EVA originally stood for European Voynich Alphabet, but later became 'Extensible Voynich Alphabet'. It was designed to be similar to Frogguy, but to use only alphabetical characters. It was designed by Gabriel Landini and myself with important contributions and suggestions from Jacques Guy. Its design was part of a larger scheme which included:

'Basic Eva' is the set of lower case characters that was identified, and 'Extended Eva' includes the representation of all types of rare characters. The basic Eva alphabet was chosen such that the transcribed text is almost pronouncible. This excellent idea from Gabriel was not to be able to 'speak Voynichese', but it makes it very easy for the human brain to recognise and remember transcribed words.

The rare characters can be classified into four categories:

  1. Unusual or rare single characters
  2. Ligatures of only 'Basic Eva' characters
  3. Ligatures including rare single characters
  4. Anything else

These were all identified in the course of the transcription exercise. As it turned out, there were three single characters that might be called unusual because they did not appear in any of the previously defined transcription alphabets, but which occurred more than 10 times in the MS. These three characters: b, u and z were assigned their own 'Basic Eva' letter (b, u and z respectively). The table of Basic Eva is included here:

Individual rare characters, or rare parts of ligatures, were assigned a 'high ASCII' code, and the way to include them into transcription files is as follows: &185; (soon to be replaced by @185;) for ascii code 185. The following image shows all such rare characters (9).

Ligatures of characters may be represented in two ways. The first is the previously mentioned capitalisation rule introduced already by Jacques Guy. As the table of basic Eva above shows, this method allows an accurate representation of the text using the Eva True Type font. A capitalised character always connects to the next one. In addition, there are the following special cases:

Often, the characters Sh, cTh, cKh, cPh and cFh are simply written as sh, cth, ckh, sph and cfh respectively, which results that their rendition in the True Type Font is not optimal.

The second way to indicate ligatures in transcription files, which is more intuitive for the human reader, is to enclose the connected characters in parentheses (soon to be changed to curly brackets). The following table shows examples of this. The script representation in this table uses the capitalisation rule.

EVA Capitalised EVA Using TTF
(ao) Ao Ao
(ee) Ee Ee
(cthh)ey cTHhey cTHhey
(ith) ITh ITh
(oy) Oy Oy
sh 1 Sh  Sh
(yk) Yk Yk

1 Example of standard though not optimal use.

Further significant efforts

The Eva alphabet, primarily in its basic form, found great reception in the community, and it was used by the Japanese Takeshi Takahashi to produce a new transcription, which was the first that could be considered almost complete. In addition, Gabriel Landini collected the important older transcriptions mentioned above into a single interlinear file, meaning that they were presented together line by line. They were all converted into Eva, since this conversion is consistent and revisersible, as mentioned above.

The brazilian Jorge Stolfi took this interlinear file and put very significant effort into improving it, adding his own transcriptions and partial transcriptions from several others, for example John Grove. As a result, this interlinear file has become the de facto source for transcription data. In the following, it will be referred to as the Stolfi-Landini interlinear file.

v101

A few years later, Glen Claston embarked on a complete transcription of the Voynich MS and devised a transcription alphabet which he called Voynich 101 (here v101 for short). The v101 alphabet was designed to keep stroke combinations that appeared to him to be single signs as single characters. Furthermore, it distinguishes between several variants of characters that were considered to be one and the same in all previous alphabets. He transcribed the entire text of the Voynich MS using this alphabet. Following is his definition of the alphabet, and allocation to ASCII values.

Also for the v101 alphabet a True Type font has been designed, and like the Eva font it allows high-quality rendition of the Voynich MS text in electronic documents (Word, Excel). Also this font can be downloaded at the site map.

Transcription file format

For the develpoment of analysis software using the various Voynich MS text transcriptions, it is highly desirable that these transcriptions are contained in files with a well-defined (standardised) format. Such files should be 'annotated', meaning that they include so-called metadata, which is not the text itself, but information about the text. This metadata should indicate for each particular piece of text on which folio or page it is located, and where it is on the page. Additional relevant information about the pages or the text may be included. Analysis software should be able to either interpret this information or to ignore it.

In the earliest days of the internet, when the transcription by Currier was being extended, a format was agreed for this new file. The result is included below. This format was extended in the course of the transcription by Gabriel Landini and myself. It was further modified by Jorge Stolfi, and used in this final version in the main resource: the Stolfi-Landini interlinear file.

The following tables explain the conventions that have been largely used in the transcription files. The transcription by Glen Claston does not follow this. It will be described briefly, further below.

# (hash) at start of line The entire line is a comment.
{ ... } All text in curly brackets is equally a comment. This may appear anywhere in a line.
< ... > This is a locus indicator, and appears at the start of each text segment. It explains where this text is to be found. The format of locus indicators is explained further below.
[ ... ] Used for alternative or uncertain reading. [ao] means it could be an a or an o, but the transcriber is not certain. The options may be separated by a | which is optional in case there are only two, but obligatory when there are more, e.g. [r|s|d]. If possible, the most likely option should be the first one.
. A word space (word separator).
, An uncertain word space.
- A drawing intrusion in the text.
= at end of line. End of a paragraph.
\ at end of line the following line is a continuation of this locus.
* A single illegible character.
? An uncertain number of illegible characters.
! A null character (only used in interlinear files, see below).

There are two types of locus indicators:

<f17r> A locus indicator without a period means the start of a new page, in this case f17r. There will be no Voynich MS text following this, but there may be a comment, with metadata referring to the entire page.
<f17r.X.nn> This is the start of a piece of text of locus type 'X'. The meaning of X is explained in the following.

In the past, I had also included a table indicating the meaning of the various locus types (X), but I had to remove it since in reality there isn't any clear definition. Gabriel Landini and I used a fairly straightforward set of identitifications, but the result has not been published so this information is not very useful. In the interlinear file, a very large number of locus types is used (more than half the alphabet), and unfortunately quite inconsistently. This situation will have to be resolved.

Jorge Stolfi, who put a great amount of effort in producing the interlinear file, introduced the useful feature where a single locus may be represented several times, as transcribed by different people. In this case the locus indicator is post-fixed by an "author ID", which is a single upper case character. It is inside the < > brackets and preceded by a semicolon, as follows:
<f17r.P.1;H> is line 1 of f17r transcribed by Takeshi Takahashi.

To (pre-)process a transcription file for statistical analysis, I developed a simple tool called VTT (Voynich Transcription Tool). This is based on the transcription file format used by Gabiel Landini and myself, and could easily be updated to work with an improved format. It allows several useful operations on the text:

It also allows selection of pages depending on 'page variables', which is a dedicated type of comment (metadata). Without going into all detail, the following line in a transcription file:
<f1r> {$L=A $H=1 $I=H}
sets the three page variables L, H and I to the values A, 1 and H respectively. (These mean respecively: Currier language = A, Currier hand = 1, Illustration type = herbal). The tool VTT can be instructed to include or exclude pages with any kind of variables set to certain values. This useful feature has also been included in a tool of Elias Schwerdtfeger, for which see below.

The v101 transcription file (for which see also below) does not use this format. It only identifies loci between angular brackets < and >, which refer to the page, a line number, or are more descriptive identification in other cases. Most of the symbols that have a special meaning in the interlinear file are actually representations of Voynich MS text characters in the v101 transcription.

Summary

To summarise, it is worth listing both the significant achievements and the remaining problems.

Achievements

By now, several independent transcriptions of the Voynich MS are available. There are three that are almost complete:

An on-going analysis shows that the "LZ" transcription, which benefited strongly from the work of Theodore Petersen, is most complete. It lacks only:

The "SL" transcription lacks about one third of the text in the Rosettes page. The completeness of the "GC" transcription is more difficult to assess, since most items that are not linear text have been grouped together. It is not 100% complete, because (for example) the label on f2r is missing. The total number of characters in the "GC" transcription, not counting spaces and annotation characters, is 158,959. This is the closest estimate we have of the number of characters in the MS. It is based on the definition of the v101 transcription alphabet (10).

The following table gives the number of loci in these transcriptions, and the estimated total number in the MS.

Transcription Nr. of loci
LZ (EVMT) 5341
SL Interlinear 5362
GC (v101) 4486
Complete MS 5385

Further details will be added as they become available.

Problem areas

Transcription of the Voynich MS is particularly difficult, because one cannot read the text. One has no help from context. The biggest problem is to decide whether two almost-similar-looking characters are two different ones, or two representations of the same one, where the variation is just caused by handwriting variations. While transcribing, one is continually forced to make such decisions, and it is unavoidable that these are subjective. A similar problem exists in deciding about word spaces. Sometimes, characters are offset just a bit and it is hard to decide whether there is a space or not. Even the introduction of the comma to represent an uncertain space doesn't completely resolve this problem. The only way out of both these problems is to have these decision made more objectively, i.e. by a piece of software (OCR) but we are far from being able to achieve this.

A second general problem is the lack of standards. The transcription files all have more or less significant differences in format. There is no place where they are all collected together. Of the interlinear file there are also several versions.

It is currently impossible to present the "GC" transcription in any of the standard formats, because its transcription alphabet includes a number of characters (e.g. # & !) that are used as annotation. A solution to this problem is already under work.

It is almost always impossible to repeat analyses done by others since one doesn't know which data they used, and there isn't really any clear way how they could have specified it. The file format(s) used nowadays are far removed from modern representation standards. Already more than a decade ago, Rafał Prinke suggested the use of TEI (Text Encoding Initiative) but it never took off. There are a few more comments about this general topic on the final page of this site.

Next-generation transcription of the text

Location of main transcription resources

Following are links to the transcription resources that are known to me. They are presented roughly in chronological order.

Notes

1
For more about the Marshall Library, see here.
2
See Reeds (1995).
3
See Bennett (1976).
4
See Brumbaugh (1978).
5
I am grateful to Jonina Duker for her recollections of these events of 40 years ago.
6
Of course, we have no way of knowing whether either assumption is correct or not.
7
On the other hand, it should be clear that not all features of a transcription made directly in Eva can be converted into the older transcription alphabets in a reversible manner.
8
Such a font has been designed by Gabriel Landini, and is used extensively at this web site. See the site map.
9
Gabriel Landini added all of them to the True Type font.
10
The question marks that appear in the GC transcription file have been counted as characters. They have been assumed to represent illegible characters.

 

Contents Home Top Prev Next
Map Refs Pics Search Pages
Copyright René Zandbergen, 2017
Comments, questions, suggestions? Your feedback is welcome.
Latest update: 22/07/2017