Page 1 of 3 1 2 3 LastLast
Results 1 to 20 of 50

Thread: ETQW oddities with glibc 2.15+ [FIX]

  1. #1

    ETQW oddities with glibc 2.15+ [FIX]

    Last edited by rorgoroth; 7th Jul 2012 at 10:07.

  2. #2

    Re: ETQW oddities with glibc 2.15+ [FIX]

    The patch.. "Courtesy of: JinxterX @ Arch Linux" - can you put that in a brighter font on your page? I like seeing my name in lights, lol

    Just kidding

  3. #3

    Re: ETQW oddities with glibc 2.15+ [FIX]

    Here's the bug report for those interested. Tried replicating it myself but couldn't even get glibc 2.15 to build.

  4. #4

    Re: ETQW oddities with glibc 2.15+ [FIX]

    Quote Originally Posted by JinxterX View Post
    The patch.. "Courtesy of: JinxterX @ Arch Linux" - can you put that in a brighter font on your page? I like seeing my name in lights, lol

    Just kidding
    Haha, well it kind of is in lights now, made a little change to point people in a more appropriate direction


    Quote Originally Posted by timestart View Post
    Here's the bug report for those interested. Tried replicating it myself but couldn't even get glibc 2.15 to build.
    Ah, I threw that up on the page just now!
    p.s - Keep trying!

  5. #5

    Re: ETQW oddities with glibc 2.15+ [FIX]

    Well, it's a compiler bug in the old version of GCC used to build ETQW for Linux which causes some localisation strings to be placed on odd (rather than even) memory addresses, thus screwing up the SSE2 optimised wide character string length function in glibc 2.15.

    If you use Archlinux the easiest thing to do is build it from ABS after modfying the PKGBUILD to include:

    Code:
    ${srcdir}/glibc/configure --prefix=/usr \
          --libdir=/usr/lib --libexecdir=/usr/lib \
          --with-headers=/usr/include \
          --enable-add-ons=nptl,libidn \
          --enable-kernel=2.6.32 \
          --with-tls --with-__thread \
          --enable-bind-now --without-gd \
          --without-cvs --disable-profile \
          --disable-multi-arch
    and then copy the new libc.so.6 into /opt/etqw.

    I also rename libgcc_s.so.1 and libstdc++.so.6 in the game dir so only the latest (distro installed) versions are used, which stops the segfault and may help with some files not being "touched" for Pro mod maps.

    The fault is not really with glibc but rather ETQW being so old.

  6. #6

    Re: ETQW oddities with glibc 2.15+ [FIX]

    For Archlinux users and glibc 2.16, sync with ABS and edit the PKGBUILD :

    Code:
    #check() {
    #  cd ${srcdir}/glibc-build
    # make -k check
    #}
    If you don't skip these checks the compile will probably fail and you waste a lot of time.

    Code:
    ${srcdir}/${pkgname}-${pkgver}/configure --prefix=/usr \
          --libdir=/usr/lib --libexecdir=/usr/lib \
          --with-headers=/usr/include \
          --enable-add-ons=nptl,libidn \
          --enable-obsolete-rpc \
          --enable-kernel=2.6.32 \
          --enable-bind-now --disable-profile \
          --enable-stackguard-randomization \
          --disable-multi-arch
    Disable the SSE optimizations to fix the string problem.

    Run makepkg.

    Then copy libc.so.6 and libm.so.6 from the pkg/lib into /opt/etqw to avoid:

    Code:
    etqw.x86: relocation error: /lib/libm.so.6: symbol __get_cpu_features, version GLIBC_PRIVATE not defined in file libc.so.6 with link time reference
    Last edited by JinxterX; 6th Jul 2012 at 21:25. Reason: fix found, typos

  7. #7
    Chief Technology Officer RR2DO2's Avatar
    Join Date
    Dec 2001
    Location
    London, UK
    Posts
    302
    Most Recent Awards:

    Re: ETQW oddities with glibc 2.15+ [FIX]

    Briefly glancing through this thread this seems like a bug in the current glibc too. If the current code tries to do an optimisation (SIMD code) that only works on memory buffers that are aligned in a certain way, it only should run that path when the memory is actually aligned.
    - Arnout 'RR2DO2'

  8. #8
    Chief Technology Officer RR2DO2's Avatar
    Join Date
    Dec 2001
    Location
    London, UK
    Posts
    302
    Most Recent Awards:

    Re: ETQW oddities with glibc 2.15+ [FIX]

    Interestingly enough it hit Sublime Text too: http://www.sublimetext.com/forum/vie...php?f=2&t=4473 . It is probably worth reopening that bug report. The risk is though that checking if a buffer is aligned or not will undo any performance gains from the optimisation.
    - Arnout 'RR2DO2'

  9. #9

    Re: ETQW oddities with glibc 2.15+ [FIX]

    Thanks for that Jinx!
    I noticed the 2.16 update the other day and thought of etqw but I haven't played the game in a while and couldn't be bothered getting the disc, luckily RR2DO2's posts caught my eye on the sidebar and got my arse in to gear, I needed to update my standalone game anyway so it's all good.

    Updated first post, site, etc.
    Also, I checked after building whilst I was sorting stuff about commenting out the checks - ****ing hell do they make it take a long time, and it failed after about 30 minutes, so thanks for that.

  10. #10

    Re: ETQW oddities with glibc 2.15+ [FIX]

    Quote Originally Posted by RR2DO2 View Post
    Interestingly enough it hit Sublime Text too: http://www.sublimetext.com/forum/vie...php?f=2&t=4473 . It is probably worth reopening that bug report. The risk is though that checking if a buffer is aligned or not will undo any performance gains from the optimisation.
    The bug report with Archlinux cannot be reopened because the fault lies within ETQW and not glibc, the Sublime Text dev post confirms that (thanks for the link, good stuff) it's up to the developer to fix these kind of problems.. "Long story short, my wchar_t strings are now all properly aligned, and everyone's happy again. The moral of the story is, of course, just use UTF-8 everywhere"

    It's conceivable glibc could be patched to work around the issue but I doubt that will happen because the cases are so rare and I can imagine glibc devs saying "it's your problem not ours", lol.

  11. #11

    Re: ETQW oddities with glibc 2.15+ [FIX]

    Quote Originally Posted by Psykorgasm View Post
    Also, I checked after building whilst I was sorting stuff about commenting out the checks - ****ing hell do they make it take a long time, and it failed after about 30 minutes, so thanks for that.
    Yep, those checks are a total waste of time in this case and they introduced even more in 2.16. My compile always fails on tst-cpuclock1, tst-cpuclock2, tst-cputimer1-3 etc which seems to be a known issue from 2.15 supposedly fixed in 2.16, but obviously not

  12. #12
    Chief Technology Officer RR2DO2's Avatar
    Join Date
    Dec 2001
    Location
    London, UK
    Posts
    302
    Most Recent Awards:

    Re: ETQW oddities with glibc 2.15+ [FIX]

    Incorrect, the wcslen specification does not have any alignment requirements. Therefor the Archlinux/glibc implementation is broken. In the Sublime Text case they took note of this broken implementation and worked around it.

    POSIX specification: http://pubs.opengroup.org/onlinepubs...ns/wcslen.html
    - Arnout 'RR2DO2'

  13. #13
    Chief Technology Officer RR2DO2's Avatar
    Join Date
    Dec 2001
    Location
    London, UK
    Posts
    302
    Most Recent Awards:

    Re: ETQW oddities with glibc 2.15+ [FIX]

    Now there is a slight grey area here. The malloc spec says "The pointer returned if the allocation succeeds shall be suitably aligned so that it may be assigned to a pointer to any type of object and then used to access such an object in the space allocated (until the space is explicitly freed or reallocated)." and in theory it could be argued that a 'string object' pointed to by a wchar pointer is 'a type of object' and thus your string objects could be expected to follow malloc defined alignment. However, this is not what the wcslen specification says and not what the wchar_t specification says; they are not required to have natural, or factor of two, alignment.

    On the other side (and you will find various best guides here and there saying this) for performance reasons and avoidance of issues like this it'd be good to be naturally aligned. ETQW doesn't enforce this on it's wide strings. Still, the fact that it crashes on an unaligned string is a bug as it is behaviour against the standard.

    It does appear as if the Linux kernel is written with a warning that you should be naturally aligned, or to expect exceptions on certain systems. If this also applies to glibc on Archlinux they need to document this, as well as make an announcement to developers when they implement a breaking change like this.
    - Arnout 'RR2DO2'

  14. #14

    Re: ETQW oddities with glibc 2.15+ [FIX]

    The normal wcslen function in glibc worked fine up until 2.15, the problem now is the automatic SSE2 optimisation of wcslen kicks in if those features are detected on the CPU. Due to the nature of the routine (written in assembler) and the size of the registers to be loaded, if a string is misaligned (i.e. at a odd memory address) then the incorrect length is returned, however, the routine is functioning 100% correctly, the fault lies with the original binary. Pretty sure there's compiler flags to ensure alignment in C++ based programs (which ETQW is).

    Take a look at wcslen-sse2.S in the glibc sources if you want, as this is the routine at the heart of the matter and it was added by a libc devs in 2.15, it's nothing to do with Archlinux. Short of bypassing this routine altogether (by disabling multi-arch entirely or introducing a patch) you have to fix any alignment stuff at source, which I assume is what the Sublime Text people did and then recompiled.

    The Linux version of ETQW however cannot be recompiled as development stopped a long time ago, therefore we have to work around the problem.

  15. #15

    Re: ETQW oddities with glibc 2.15+ [FIX]

    Quote Originally Posted by JinxterX View Post
    The normal wcslen function in glibc worked fine up until 2.15, the problem now is the automatic SSE2 optimisation of wcslen kicks in if those features are detected on the CPU. Due to the nature of the routine (written in assembler) and the size of the registers to be loaded, if a string is misaligned (i.e. at a odd memory address) then the incorrect length is returned, however, the routine is functioning 100% correctly, the fault lies with the original binary. Pretty sure there's compiler flags to ensure alignment in C++ based programs (which ETQW is).

    Take a look at wcslen-sse2.S in the glibc sources if you want, as this is the routine at the heart of the matter and it was added by a libc devs in 2.15, it's nothing to do with Archlinux. Short of bypassing this routine altogether (by disabling multi-arch entirely or introducing a patch) you have to fix any alignment stuff at source, which I assume is what the Sublime Text people did and then recompiled.

    The Linux version of ETQW however cannot be recompiled as development stopped a long time ago, therefore we have to work around the problem.
    I disagree. Optimizations of this nature should only be enabled if 1) the hardware supports the features used and 2) the input data is valid for the optimization.

    If _only_ #2 were not true - life for cross platform devs would be just that bit easier

    Also - compilers cannot ensure alignment of dynamic data.
    - eQ Domipheus

  16. #16

    Re: ETQW oddities with glibc 2.15+ [FIX]

    It's true that wcslen in glibc doesn't check for misaligned data before going into the SSE2 optimisation, I'm not sure why, maybe they just assume developers use memalign() to a 16 byte boundary in their code, or write their own small function to dynamically allocate aligned memory suitable for SSE2 registers? I know.. these are crazy, wacky ideas huh?

  17. #17

    Re: ETQW oddities with glibc 2.15+ [FIX]

    NOOB question ALERT

    is this what running games on linux involve? Just curious, dont kill me.

  18. #18

    Re: ETQW oddities with glibc 2.15+ [FIX]

    Quote Originally Posted by Gradis View Post
    NOOB question ALERT

    is this what running games on linux involve? Just curious, dont kill me.
    I wouldn't worry about any bodily harm. They obviously don't have the time for any of that.

    Actually, I'm pretty sure they're making most of this stuff up.

  19. #19

    Re: ETQW oddities with glibc 2.15+ [FIX]

    Quote Originally Posted by JinxterX View Post
    It's true that wcslen in glibc doesn't check for misaligned data before going into the SSE2 optimisation, I'm not sure why, maybe they just assume developers use memalign() to a 16 byte boundary in their code, or write their own small function to dynamically allocate aligned memory suitable for SSE2 registers? I know.. these are crazy, wacky ideas huh?

    Assumptions. The bane of software development!
    Last edited by Domipheus; 8th Jul 2012 at 22:40. Reason: Spelling? WHO NEEDS IT :)
    - eQ Domipheus

  20. #20

    Re: ETQW oddities with glibc 2.15+ [FIX]

    Quote Originally Posted by rorgoroth View Post
    Sorry for necromancy. But glibc related crashes makes the game unbearable on Linux. Unfortunately, those Dropbox uploads are down at the moment. Can they be reuploaded please? Thank you.

    Regards.

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •