nvrtc.nvrtcCompileProgram is changing the preferred encoding from UTF-8 to ANSI_X3.4-1968

Dear developers,

I found out that calling the NVRTC for compilation is changing the preferred encoding for the current Python instance.

For more details and to reproduce the issue, please refer to this StackOverflow question.

Do you have an idea on why this happens, and how it is possible to revert the preferred encoding to its original setting?

Thank you in advance

This sounds like an issue of NVRTC rather than CUDA Python. The issue was also reproducible in CuPy built without CUDA Python.

>>> import locale, cupy
>>> locale.getpreferredencoding()
'UTF-8'
>>> cupy.arange(10)
array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
>>> locale.getpreferredencoding()
'ANSI_X3.4-1968'

Env: CUDA 11.8 / Ubuntu 20.04

Good to know, I have filed a bug to Nvidia now, let's see.
Thank you again.

Just to give a small update. By discussing the issue with Nvidia we found out that it is possible to export LC_ALL="POSIX" as a workaround to avoid NVCC changing the encoding to ASCII.

The causes of the bug are still unknown and I will report when I have any other news.

Since this is a bug outside of CUDA Python, I'll close this issue.

Thanks for sharing that workaround. If there's a link you can share for where this bug is being tracked, I'm sure folks would appreciate it.

Here is the link to the bug report: https://developer.nvidia.com/nvidia_bug/3833924

Thank you again for your help

Fixes #625, which is due to a bug caused by NVIDIA (NVIDIA/cuda-pytho…

Explicit file encoding for cuda mixed environment due to issue NVIDIA…

Just to give a small update. By discussing the issue with Nvidia we found out that it is possible to export LC_ALL="POSIX" as a workaround to avoid NVCC changing the encoding to ASCII.

The causes of the bug are still unknown and I will report when I have any other news.

Any updates here? It is indeed a bug in NVRTC, specifically nvrtcCompileProgram and can be reproduced in C++:

#include <langinfo.h>
#include <cuda.h>
#include <nvrtc.h>
#include <vector>
#include <iostream>

int main(){
    setlocale(LC_ALL, "");
    std::string code = "";
    std::vector<const char*> args = {"--gpu-architecture=sm_80"};
    nvrtcProgram program;
    nvrtcCreateProgram(&program, code.c_str(), nullptr, 0, nullptr, nullptr);
    std::cout << nl_langinfo(CODESET) << '\n';
    nvrtcCompileProgram(program, args.size(), args.data());
    std::cout << nl_langinfo(CODESET) << '\n';
}

Compile with nvcc test.cu -lnvrtc and observe:

$ LC_ALL=en_US.UTF-8 ./a.out 
UTF-8
ANSI_X3.4-1968
$ LC_ALL=C.UTF-8 ./a.out 
UTF-8
ANSI_X3.4-1968
$ LC_ALL=POSIX ./a.out 
ANSI_X3.4-1968
ANSI_X3.4-1968

I'm a bit confused about the statement "export LC_ALL="POSIX" as a workaround to avoid NVCC changing the encoding to ASCII" as that by definition sets the encoding to ASCII in the first place. So the change is only masked.

Explicit file encoding for cuda mixed environment due to issue NVIDIA…

@redsnic

@redsnic Any news about this? The upstream bug report seems to be access-restricted?

Sorry for the late reply.

This was the last update:

We are glad to let you know our compiler team has fixed the issue . nvrtc will no longer change user program locale/encoding . This fix will aim a future CUDA 12.x release around September 2024 . Feel free to comment back here if you need help in confirming a bug fix version in the future.

However, the bug is still open. So I don't know if it was fixed yet.

@redsnic

@redsnic thank you! From what I have seen elsewhere; the fix was indeed pushed, but only targets CUDA 12.7+. In the meantime, using Python in UTF-8 mode allows bypassing it (the locale is still changed, but Python doesn't take it into account.

Merging EE-ADC code (#38)

Molecular ADC for electronic excitations (EE-ADC) (#2893)

Merging EE-ADC code (#38)

Navigation Menu

Repository navigation

nvrtc.nvrtcCompileProgram is changing the preferred encoding from UTF-8 to ANSI_X3.4-1968 #29

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

nvrtc.nvrtcCompileProgram is changing the preferred encoding from UTF-8 to ANSI_X3.4-1968 #29

Description

Activity

kmaehashi commented on Oct 15, 2022

redsnic commented on Oct 15, 2022

redsnic commented on Oct 27, 2022

vzhurba01 commented on Dec 5, 2022

redsnic commented on Dec 6, 2022

Flamefire commented on Mar 7, 2024

LoicGrobol commented on Feb 17, 2025

redsnic commented on Feb 24, 2025

LoicGrobol commented on Feb 24, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions