Ciarán's free software notes
Ciaran O'Riordan's irregularly kept software freedom journal
Limit entries displayed: [ 2 ] [ 4 ] [ 6 ] [ 8 ]
Gettext for static websites
Here's how I implemented a translation management system for a static website, using GNU gettext. For the impatient, I've distilled it to 11 instructions at the end.
Goal
This system allows block-by-block translation (string-by-string), which is better than page-by-page because:
- Changes to non-translated parts will be applied to all translations automatically (formatting, tags, images, maybe dates, names, links, etc.).
- By storing the text blocks of all pages together, repeated blocks will only have to be translated once (menu text, copyright notices, headings, etc.).
- You won't get lost when the original changes while the translation is still in progress.
- When you change a paragraph in the original, it's easier to see what parts of the translations need to be updated.
For such a system, the abstract steps are:
- Somehow mark each translatable text block in your webpage. The non-translatable parts will become a shared frame.
- Extract the blocks into a database. Translate.
- Find or write some software to merge the blocks back into the frame to remake the original webpage - but with the option of taking the text blocks from either the English database or one of the translated versions of the database.
Gettext seemed like an obvious possibility, and everything's working perfectly now, but it took me eight hours. The difficulty was that the existing documentation is all geared toward using gettext for computer programs, not for websites or documents. That's when I realised that I must document what I did:
What I did
I started by minimally turning my webpage into a computer program. This involved five steps:
- Write a tiny program that prints some text (a string) into a file.
- Copy the webpage into the program in place of the string.
- Insert some standard bits of code required by gettext.
- Break the string into smaller strings, separating translatable from non-translatable
- Mark the translatable strings with gettexts' tag (the format of the tags depend on which programming language you use but it's usually something involving an underscore _ ).
Gettext works with lots of programming languages, so take your pick
from the examples that come with the package. On my computer, these
are in this
folder:
/usr/share/doc/gettext-doc/examples/
The choice of language isn't important. The code will be dead
simple.
Here's my original index.html:
<html>
<head>
<title>Cow</title>
</head>
<body>
<p>See also: <a href="http://fsfe.org/">FSFE</a></p>
</body>
</html>
Of the supported programming languages, I
choose Scheme (a
dialect of Lisp). At first glance, the code below looks complex,
but you'll only have to modify the first and third chunks. The
first chunk defines three variables which should be
self-explanatory. All the webpage text is in the third chunk. It's
broken up into blocks and I've put gettext tags for
Scheme (_ ) around the translatable blocks. Here
it is, generate-index.scm:
#!/usr/bin/guile -s
!#
(define output-filename "index.html")
(define project-name "ciarans-website")
(define build-directory "/home/ciaran/website-build/")
(use-modules (ice-9 format))
(catch #t (lambda () (setlocale LC_ALL "")) (lambda args #f))
(textdomain project-name)
(bindtextdomain project-name build-directory)
(define _ gettext)
(define page-text (string-append
"<html><head>\n<title>"
(_ "Cow")
"</title>\n</head>\n<body>\n<p>"
(_ "See also: ")
"<a href=\"http://fsfe.org/\">"
(_ "FSFE")
"</a></p>\n"
"</body></html>\n\n"))
(define the-file (open-file output-filename "w"))
(display page-text the-file)
Three of the eight strings are marked as translatable. The other five are part of the shared frame that will be the same no matter what language version of the page is being generated.
Remember to replace any quote marks in your HTML with
backslash-quote (\"), and to add a few line breaks
(\n) to make the output readable. Those are the quote
and the newline sequences for Scheme. They're the same in a few
other languages, but they're different in others.
Before you continue, you must set the "build-directory"
variable to the directory where generate-index.scm is.
If you don't, everything will seem to work but your program will
never access the translated strings.
That done, you extract the translatable strings with these two commands:
$ xgettext --language=scheme -d ciarans-website -k_ generate-index.scm$ mv ciarans-website.po ciarans-website.pot
And then you can create a file (a "po" file) for French translations with this command:
$ msginit --locale=fr
One part of the gettext manual says that
"msginit" is optional - that you can do it
manually instead, but this didn't work for me at all. I spent two
hours diagnosing that problem. Use msginit.
This creates fr.po which you can edit with any text
editor. There will be a line at the top like this:
"Content-Type: text/plain; charset=UTF-8\n"
If your charset is "ASCII", you should probably change it to UTF-8. If your charset is something else and you get error messages from other gettext tools (such as msgmerge) about invalid characters, then changing charset to UTF-8 might also be the answer. There'll also be a field for content-transfer-encoding. The manual says that should always be "8bit".
Emacs is particularly good for editing po files because it has a special editing mode for them.
Next you have to convert your po file into the special mo format and put it in the subdirectory where gettext expects it to be with these two commands:
$ mkdir -p fr/LC_MESSAGES$ msgfmt --output-file=fr/LC_MESSAGES/ciarans-website.mo fr.po
Make the Scheme file executable, and that's it!
ciaran@hide:~/tests/simple-page$ LANGUAGE=fr ./generate-index.scm; cat index.html
<html><head>
<title>Vache</title>
</head>
<body>
<p>Voir aussi : <a href="http://fsfe.org/">La FSFE</a></p>
</body></html>
ciaran@hide:~/tests/simple-page$ LANGUAGE=en ./generate-index.scm; cat index.html
<html><head>
<title>Cow</title>
</head>
<body>
<p>See also: <a href="http://fsfe.org/">FSFE</a></p>
</body></html>
ciaran@hide:~/tests/simple-page$
Ok, so there's your proof-of-concept. Next, I have to convert my
site to this system and maintain it (using msgmerge).
I'll try to keep notes to publish here. Lastly, thanks to the ILUG
community,
who suggested
some alternatives.
The instructions
- Make an empty file
generate-index.scm - Copy my
generate-index.scm(above) into your file - Adjust the build-directory (3rd defined variable) in
generate-index.scmto point to the directory where yourgenerate-index.scmis $ xgettext --language=scheme -d ciarans-website -k_ generate-index.scm$ mv ciarans-website.po ciarans-website.pot$ msginit --locale=fr- edit
fr.poto add translations of the three text strings $ mkdir -p fr/LC_MESSAGES$ msgfmt --output-file=fr/LC_MESSAGES/ciarans-website.mo fr.po$ chmod +x generate-index.scm$ LANGUAGE=fr ./generate-index.scm; cat index.html
--
Ciarán O'Riordan, (RSS)
Support free software: Join FSFE's
Fellowship
Status of FSFE's legal dept: FTF
Inside FSFE, we talk a lot about our legal department, the FTF. I was in the Zurich office a while ago with the FTF's coordinator, Shane Coughlan, and took the opportunity to gather some info for anyone interested.
The FTF works in five main areas:
- Building a European legal network
- Producing documentation
- GPL enforcement
- FLA agreement for copyright management
- Training and consultation
Legal network
For two years now, Shane's been building this network of lawyers and licence experts which now includes 145 members. Three quarters of the members are lawyers. The others are experts in company policy, licences, or technical aspects of licence enforcement. About 120 come from Europe, and the others are spread across The Philippines, Japan, Singapore, China, Taiwan, South Korea, Australia, Canada, and the USA.
I think most network members support FSFE's work. I know others don't, and that's ok. Whether we share a vision for the future or not, everyone has to obey the GPL, so it's useful for free software licence compliance specialists to talk with each other and share best practices. Most discussion is about the GPL, but we talk about all free software licences.
Other topics such as software patents and antitrust are also discussed.
FTF organised Europe's first free software legal conference in Amsterdam in April with 53 members of the legal network attending - 5 coming from outside of Europe. Feedback from the attendees was very positive, so there'll certainly be more such conferences in the mid-term future.
Producing documentation
Some of the documentation produced by FTF is already online - there's a documentation section on the website. One type is the documents Shane produces, such as the useful tips for users of GPL, and same for vendors of GPL'd software.
Another type is the documents that the network members pass around among themselves. These are usually procedures, guidelines and other documents that have existed for a long time internally in their various companies. If these documents cannot be published, then at least by circulating them privately in the network, they can expand the knowledge of many free software lawyers. In the long term, some of these documents might become publishable or, if allowed by the author(s), will be used for the basis of our own documents.
GPL enforcement
This is the quietest part of FTF's work. We don't go to court, and we don't go to Slashdot. Compliance is gotten while maintaining relations with the distributor. Of course, we also work with gpl-violations.org, which does take people to court. We've been working with them since FTF started in 2006, and earlier this year we agreed to deepen that relationship. And to reduce problems originating from the manufacturers, the "for users" and "for vendors" useful tips have been translated and distributed in Chinese and Korean (they're temporarily offline during a webpage reorganisation).
FLA: Fiduciary Licence Agreement
The Fiduciary Licence Agreement is used when a developer wants to grant an organisation the ability to enforce the licence of the code, and give them the ability to update the licence of the code - with the limit that the new licence must also be a free software licence. The developer doesn't lose their copyright, so they can also enforce and change the licence of their code, without limits.
This is important for legal maintainability of a project. If a problem is discovered with the licence, having a central body maintaining the copyright would allow the project to avoid the difficulty of find all past authors and getting unanimous agreement on what changes to make to the licence. A big recent success is that KDE announced that they're going to use it. More good news is that it should soon be in 10 languages. FSFE has become the legal guardian for a small number of projects, such as OpenSwarm and Bacula, but that's not our focus for the FLA.
Training and consulation
Lastly, Shane visits companies or regions to deliver training courses on free software licences and legal issues. A skeleton course is online on the SELF platform: The strategic implementation of Free Software in business.
These courses are educational for legal experts dealing with free software, and they're also a way for the FTF to be financially sustainable. If you work for a company that has a legal department and that deals with free software, you can help the FTF by suggesting to your boss that they get Shane to deliver a course. With so many big companies profiting from free software, and with the FTF providing value to so many lawyers for free, it's only right that the costs of the FTF be covered by these companies rather than the income we get from the community (through the Fellowship). Another significant source of funding for which we're grateful is NLnet.
Ok. That's what I see the FTF doing. If you read this far, I hope I answered some questions!
--
Ciarán O'Riordan,
Support free software: Join FSFE's
Fellowship
