[RVGL] Localization support

Report issues with v1.2 here.

Moderators: Huki, jigebren

[RVGL] Localization support

sebr
Methanolic
sebr
Methanolic
Joined: 29 Jul 2010, 05:26 PM

15 Aug 2015, 11:50 PM #1

rvgl can't deal with : é è à... on folder names...

an exemple of car who work on rv 1.0 & 1.2 but not with rvgl : ToyàPig
Quote
Like
Share

sebr
Methanolic
sebr
Methanolic
Joined: 29 Jul 2010, 05:26 PM

16 Aug 2015, 03:57 PM #2

Huki @ 16 Aug 2015, 03:41 PM wrote:
sebr wrote:rvgl can't deal with : é è à... on folder names...
an exemple of car who work on rv 1.0 & 1.2 but not with rvgl : ToyàPig
Hmm, on Windows it can't deal with the folder names? Is the car not even detected, or are you able to select it, but you get "can't load ... prm" errors when the car is being loaded?
Btw, I downloaded the car (a few minutes before you removed it :P) but I'll get into more details in a separate thread (I'll split your post).
on rvgl i can select the car but i get the message that revolt can't load the bmp
and the car folder name is different on the error message : toyàpig become toy?pig

note : i'm in trouble with bropbox, lost all my files :( (lucky me i have another backup, so i hope i can fix my dropbox soonly)
Quote
Like
Share

Huki
... as a Newt
Huki
... as a Newt
Joined: 30 Dec 2006, 04:01 PM

18 Aug 2015, 10:50 AM #3

I tested on Windows and it's clear that the models and hul are loaded successfully in rvgl, but the bmp fails with your error message. Incidentally, on Linux I can't load any of the car's files, neither with RVGL nor with v1.2 under Wine.

As far as rvgl on Windows is concerned, the problem is that we use SDL to load the bitmap, and it takes the file path in the standard UTF-8 encoding. But re-volt uses an extended Ascii encoding that is specific to Windows (called the Windows-1252). The language strings and the parameters.txt files are all supposed to be saved in this encoding. Likewise, the folder names in Windows should use the same encoding (which it does by default), so that there is no mismatch between the path in the parameters file and the actual path in Windows.

So when we pass the ToyàPig path from the parameters file (in Windows-1252 encoding) to SDL, it assumes it's a UTF-8 path and does some internal conversion to Windows specific encoding before opening the file - but the path is already in Windows specific encoding in the first place.. :wacko:
We can try to workaround this by re-saving the parameters.txt file in UTF-8 encoding, but now the prms and hull which we are loading directly will fail (because this is not the encoding used by Windows file names). And of course, the name of the car displayed will be wrong too, because re-volt itself only supports the 1252 character set...

The story in Linux is entirely different: here UTF-8 is universally used (by default), even to encode the file names on the system. This is very clean and straightforward, and considering that SDL also expects UTF-8, we should update RVGL too to use the same standard. It means:
  • Language and Parameters text files can be provided with UTF-8 encoding: this will open up the possibility to support numerous new languages, both for file / folder names and in-game, that we can't support right now with extended ascii.
  • Text files using the legacy 1252 encoding will be automatically converted to UTF-8 when loading.
  • All file paths can then be used directly on Linux, and converted to appropriate encoding on Windows.
Back to our original problem, we can bring back the old behavior on Windows by converting just the car texture path to UTF-8 before sending it to SDL, or we can load the texture ourselves to memory and let SDL to the necessary processing (I don't like this because it would take double the texture's size in memory during loading time and also delay the loading time). But maybe an urgent fix is not required: we can just wait till we switch over entirely to UTF-8.
Quote
Like
Share

sebr
Methanolic
sebr
Methanolic
Joined: 29 Jul 2010, 05:26 PM

18 Aug 2015, 01:30 PM #4

as you said "an urgent fix is not required"

just keep it in mind and found the best fix B)

note : i didn't remember having trouble to load this car on linux with rv 1.2... but i didn't go back to linux ince a long time, i'll check that as soon as i can
Quote
Like
Share

sebr
Methanolic
sebr
Methanolic
Joined: 29 Jul 2010, 05:26 PM

21 Aug 2015, 12:28 PM #5

just tested on Linux Ubuntu 14.04 (32bit)
and Wine 1.6.2 set for Windows XP and imported dll from win xp : d3d9_36.dll, dplayx.dll, dpnet.dll, dpnhpast.dll, dpwsockx.dll

with alpha 15.0418 ToyàPig work as fine as on win 8
and tracks with instances named like mound.PRM work fine too

with rvgl 15.0814a choosing ToyàPig = "can't load mipmap texture..." message then crash
and tracks with instances named like "mound.PRM" or "MOUND.PRM" don't load this instance, it must be renamed as "mound.prm"... but the associated ncp always work if prm is loaded

SuperMarket 3 is a good test candidat

note i didn't tryed linux rvgl
Quote
Like
Share

Huki
... as a Newt
Huki
... as a Newt
Joined: 30 Dec 2006, 04:01 PM

25 Aug 2015, 07:24 PM #6

sebr @ 21 Aug 2015, 05:58 PM wrote:just tested on Linux Ubuntu 14.04 (32bit)
and Wine 1.6.2 set for Windows XP and imported dll from win xp : d3d9_36.dll, dplayx.dll, dpnet.dll, dpnhpast.dll, dpwsockx.dll
with alpha 15.0418 ToyàPig work as fine as on win 8
Hmm can you check the locale setting in your Linux? I can get it by typing "locale" at the terminal, my output:

Code: Select all

locale
LANG=en_IN
LANGUAGE=en_IN:en
LC_CTYPE="en_IN"
LC_NUMERIC=en_IN.UTF-8
LC_TIME=en_IN.UTF-8
LC_COLLATE="en_IN"
LC_MONETARY=en_IN.UTF-8
LC_MESSAGES="en_IN"
LC_PAPER=en_IN.UTF-8
LC_NAME=en_IN.UTF-8
LC_ADDRESS=en_IN.UTF-8
LC_TELEPHONE=en_IN.UTF-8
LC_MEASUREMENT=en_IN.UTF-8
LC_IDENTIFICATION=en_IN.UTF-8
LC_ALL=
Quote
Like
Share

sebr
Methanolic
sebr
Methanolic
Joined: 29 Jul 2010, 05:26 PM

25 Aug 2015, 08:02 PM #7

Code: Select all

LANG=fr_FR.UTF-8
LANGUAGE=fr:en
LC_CTYPE="fr_FR.UTF-8"
LC_NUMERIC=fr_FR.UTF-8
LC_TIME=fr_FR.UTF-8
LC_COLLATE="fr_FR.UTF-8"
LC_MONETARY=fr_FR.UTF-8
LC_MESSAGES="fr_FR.UTF-8"
LC_PAPER=fr_FR.UTF-8
LC_NAME=fr_FR.UTF-8
LC_ADDRESS=fr_FR.UTF-8
LC_TELEPHONE=fr_FR.UTF-8
LC_MEASUREMENT=fr_FR.UTF-8
LC_IDENTIFICATION=fr_FR.UTF-8
LC_ALL=
Quote
Like
Share

Huki
... as a Newt
Huki
... as a Newt
Joined: 30 Dec 2006, 04:01 PM

26 Aug 2015, 12:58 PM #8

Strange that you use UTF-8 too, but when I extract the ToyáPig.zip archive on my Linux, this is what I get:

As you can see, instead of using the Ascii value for á it uses the Unicode character 0080 (in hex) which is a non-displayable character code and it doesn't work with v1.2a (under Wine). I can edit the folder name to use the proper á character, but this is still saved in UTF-8 encoding, not Ascii. There is no way this can work because the paths in the Parameters.txt file are saved in Ascii encoding, so trying to open those paths will fail (i.e., ToyáPig folder with Ascii á doesn't exist in the hard drive).

Maybe you are using a shared NTFS partition for re-volt? I mean, you extracted the ToyáPig folder in Windows, then booted into Linux and mounted that NTFS partition. Perhaps in that case the Ascii name is preserved.. I'm still confused. :wacko:
Quote
Like
Share

sebr
Methanolic
sebr
Methanolic
Joined: 29 Jul 2010, 05:26 PM

26 Aug 2015, 06:12 PM #9

Huki @ 26 Aug 2015, 01:58 PM wrote: Maybe you are using a shared NTFS partition for re-volt? I mean, you extracted the ToyáPig folder in Windows, then booted into Linux and mounted that NTFS partition. Perhaps in that case the Ascii name is preserved.. I'm still confused.  :wacko:
it's exactly what i do
on the same computer i can use win8 or linux
and revolt is on my ntfs partition
I set linux to automaticaly mount ntfs partition
this way i only have 1 re-volt folder that i can use on win8 and on linux
all revolt updates are made from win8

(note: it's not ToyáPig but ToyàPig, i don't think it change anything :P )
Quote
Like
Share

Huki
... as a Newt
Huki
... as a Newt
Joined: 30 Dec 2006, 04:01 PM

25 Feb 2016, 06:16 PM #10

I've added the support for international characters in file paths. Basically, filenames like ToyàPig will now load correctly in both Windows and Linux.
In addition, on Linux we are now able to load filenames with any valid UTF-8 symbol (Russian, Greek, etc). In-game display is still limited to extended Ascii characters, so if you actually set eg, a Greek name for the car, it'll be displayed as question marks "????". UTF-8 support is only for the filenames (and the path entries in parameters.txt).
See the screenshots below for an example:
foldername_utf8
parampath_utf8 (result)
paramname_utf8 (result)

The above is an interesting first step towards full internationalization support for RVGL. It's not an immediately useful feature at present (especially since it's only for Linux), but it helps confirm what is possible. We are using libunistring on both OS.

I'm attaching 2 builds for Windows and Linux. Can you test on both OS and see if ToyàPig loads correctly?
NOTE: as usual, on Linux you need to run the rename script to rename the folder and texture to lowercase, as "toyàpig". Btw, the current rename script only renames A-Z, it doesn't handle diacritics or UTF-8 symbols. I've updated the script now, but it doesn't matter as the à in your car is already lower.

Download:
rvgl_filepath_test_win32
rvgl_filepath_test_linux
Quote
Like
Share

sebr
Methanolic
sebr
Methanolic
Joined: 29 Jul 2010, 05:26 PM

25 Feb 2016, 07:35 PM #11

tested on windows : it's work B)
but if i choose a custom car i can't go back to stock cars (and CARNIVAL don't help)

I'll try linux version this week-end
Quote
Like
Share

Huki
... as a Newt
Huki
... as a Newt
Joined: 30 Dec 2006, 04:01 PM

26 Feb 2016, 04:28 PM #12

sebr @ 26 Feb 2016, 01:05 AM wrote: tested on windows : it's work B)
but if i choose a custom car i can't go back to stock cars (and CARNIVAL don't help)
Very strange :blink: Even the rookie cars are locked?
I tried:
1) select custom car and go to car preview, then go back to car select.
2) select custom car, start a race and come back to frontend.
Either way I can select a stock car (don't get the bug). But I tested in Linux... I'll try booting into Windows later, but maybe you can take video / screenshot of the situation?
Quote
Like
Share

revolting
Alcoholic
revolting
Alcoholic
Joined: 27 Sep 2013, 08:01 AM

27 Feb 2016, 08:55 AM #13

haha, I didn't even find where to download such a car lol
AKA geceb
Quote
Like
Share

sebr
Methanolic
sebr
Methanolic
Joined: 29 Jul 2010, 05:26 PM

27 Feb 2016, 11:55 AM #14

where is saved the last car used on RVGL win ?
- I deleted all custom car and the only one i can use is "mistery"
- deleted all files in Profiles folder and at restart it ask me to create a profile name but car selection still locked on "mistery"
- if i delete "mistery" car it still load a dummy unvisible car :blink:

If i go back to previouse RVGL (on the same folder) car selection work perfectly and leave it to "volken turbo"

I put back this patch and ... only "mistery" or custom car can be selected :wacko:
Quote
Like
Share

Huki
... as a Newt
Huki
... as a Newt
Joined: 30 Dec 2006, 04:01 PM

27 Feb 2016, 01:52 PM #15

sebr @ 27 Feb 2016, 05:25 PM wrote: where is saved the last car used on RVGL win ?
- I deleted all custom car and the only one i can use is "mistery"
- deleted all files in Profiles folder and at restart it ask me to create a profile name but car selection  still locked on "mistery"
- if i delete "mistery" car it still load a dummy unvisible car  :blink:

If i go back to previouse RVGL (on the same folder) car selection work perfectly and leave it to "volken turbo"

I put back this patch and ... only "mistery" or custom car can be selected  :wacko:
Ah, I think I found the bug.. wait for a fixed build..

EDIT: fixed, try downloading the build again.
Quote
Like
Share

sebr
Methanolic
sebr
Methanolic
Joined: 29 Jul 2010, 05:26 PM

27 Feb 2016, 03:26 PM #16

Huki @ 27 Feb 2016, 02:52 PM wrote:
sebr @ 27 Feb 2016, 05:25 PM wrote: where is saved the last car used on RVGL win ?
- I deleted all custom car and the only one i can use is "mistery"
- deleted all files in Profiles folder and at restart it ask me to create a profile name but car selection  still locked on "mistery"
- if i delete "mistery" car it still load a dummy unvisible car   :blink:

If i go back to previouse RVGL (on the same folder) car selection work perfectly and leave it to "volken turbo"

I put back this patch and ... only "mistery" or custom car can be selected  :wacko:
Ah, I think I found the bug.. wait for a fixed build..

EDIT: fixed, try downloading the build again.
works fine on windows now
- load toyàpig as Ai-car or race car
- read old profiles folder from original rvgl (previous test build didn't)
- CARNIVAL and search allow again stock cars B)

will test on linux as soon as possible

EDIT : just tested on linux, works as on last windows test build :)
Quote
Like
Share