Skip to content

Add torchaudio transducer loss - #1

Open
astaff wants to merge 4 commits into
developfrom
add-torchaudio-transducer-loss
Open

Add torchaudio transducer loss#1
astaff wants to merge 4 commits into
developfrom
add-torchaudio-transducer-loss

Conversation

@astaff

@astaff astaff commented Jun 2, 2021 •
edited
Loading

Copy link
Copy Markdown
Owner

Note: call to rnnt_loss aborts with

terminate called after throwing an instance of 'c10::Error'
  what():  cannot call get_autograd_meta() on undefined tensor
Exception raised from get_autograd_meta at /pytorch/torch/csrc/autograd/variable.cpp:318 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x42 (0x7f3b7fa53402 in /home/ubuntu/.env/lib/python3.8/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, char const*) + 0x5f (0x7f3b7fa4fe6f in /home/ubuntu/.env/lib/python3.8/site-packages/torch/lib/libc10.so)
frame #2: <unknown function> + 0x35a8dab (0x7f3b83081dab in /home/ubuntu/.env/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)

when using values from unit test it crashes unless I do logits.requires_grad_(True). Doing so with actual input still leads to crash.

@astaff

astaff commented Jun 2, 2021

Copy link
Copy Markdown
Owner Author

Seems like log_probs.requires_grad_(True) makes things not crash.

The code below seems to be reproducing the error:

from torchaudio.prototype.rnnt_loss import RNNTLoss
import torch


blank_index = -1
rnnt_loss = RNNTLoss(blank=blank_index, fused_log_softmax=False)
_log_probs = torch.tensor(
    [
        [
            [
                [0.1, 0.6, 0.1, 0.1, 0.1],
                [0.1, 0.1, 0.6, 0.1, 0.1],
                [0.1, 0.1, 0.2, 0.8, 0.1],
            ],
            [
                [0.1, 0.6, 0.1, 0.1, 0.1],
                [0.1, 0.1, 0.2, 0.1, 0.1],
                [0.7, 0.1, 0.2, 0.1, 0.1],
            ],
        ]
    ],
    dtype=torch.float,
)
_targets = torch.tensor([[1, 2]], dtype=torch.int)
_input_lens = torch.tensor([2], dtype=torch.int)
_target_lens = torch.tensor([2], dtype=torch.int)

log_probs = _log_probs.to(device="cuda")
targets = _targets.to(device="cuda")
input_lens = _input_lens.to(device="cuda")
target_lens = _target_lens.to(device="cuda")

# log_probs.requires_grad_(True)

loss_value = rnnt_loss(
    logits=log_probs,
    targets=targets,
    logit_lengths=input_lens,
    target_lengths=target_lens,
)

print(loss_value.mean())

@vincentqb

vincentqb commented Jun 17, 2021

Copy link
Copy Markdown

occurs both on cpu and gpu

from torchaudio.prototype.rnnt_loss import RNNTLoss
import torch

blank_index = -1
rnnt_loss = RNNTLoss(blank=blank_index, fused_log_softmax=False)

log_probs = torch.tensor(
    [
        [
            [
                [0.1, 0.6, 0.1, 0.1, 0.1],
                [0.1, 0.1, 0.6, 0.1, 0.1],
                [0.1, 0.1, 0.2, 0.8, 0.1],
            ],
            [
                [0.1, 0.6, 0.1, 0.1, 0.1],
                [0.1, 0.1, 0.2, 0.1, 0.1],
                [0.7, 0.1, 0.2, 0.1, 0.1],
            ],
        ]
    ],
    dtype=torch.float,
)
targets = torch.tensor([[1, 2]], dtype=torch.int)
input_lens = torch.tensor([2], dtype=torch.int)
target_lens = torch.tensor([2], dtype=torch.int)

# log_probs.requires_grad_(True)

loss_value = rnnt_loss(
    logits=log_probs,
    targets=targets,
    logit_lengths=input_lens,
    target_lengths=target_lens,
)

print(loss_value.mean())

When requires_grad=True the user tells pytorch that they want the gradient to be computed. When requires_grad=False, we are telling pytorch that we don't care about gradients with respect to those variables. This can be used to freeze weights.

In that case, the code simply creates a null pointer tensor here for the gradient. This needs further investigation, but it appears this initialized tensor is still accessed later, causing an error.

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-1-4e67278199cb> in <module>
     28 # log_probs.requires_grad_(True)
     29 
---> 30 loss_value = rnnt_loss(
     31     logits=log_probs,
     32     targets=targets,

~/anaconda3/envs/audio-built/lib/python3.9/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1052         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1053                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1054             return forward_call(*input, **kwargs)
   1055         # Do not call functions when jit is used
   1056         full_backward_hooks, non_full_backward_hooks = [], []

~/anaconda3/envs/audio-built/lib/python3.9/site-packages/torchaudio/prototype/rnnt_loss.py in forward(self, logits, targets, logit_lengths, target_lengths)
    100             target_lengths (Tensor): Tensor of dimension (batch) containing lengths of targets for each sequence
    101         """
--> 102         return rnnt_loss(
    103             logits,
    104             targets,

~/anaconda3/envs/audio-built/lib/python3.9/site-packages/torchaudio/prototype/rnnt_loss.py in rnnt_loss(logits, targets, logit_lengths, target_lengths, blank, clamp, fused_log_softmax, reuse_logits_for_grads)
     45         blank = logits.shape[-1] + blank
     46 
---> 47     costs, gradients = torch.ops.torchaudio.rnnt_loss(
     48         logits=logits,
     49         targets=targets,

RuntimeError: cannot call get_autograd_meta() on undefined tensor

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

See More

Successfully merging this pull request may close these issues.

2 participants

@astaff @vincentqb