Conversion error

I noticed that the conversion didn't give the same result depending on the machine running the ConvertAPI PHP library. I found that this was due to the encoding when the file was uploaded.

I haven't investigated this in depth but it's a possibility. I managed to correct the problem by changing the header to ‘Transfer-Encoding: gzip,deflate’.

File:
lib/ConvertApi/Client.php
Line:
30
Original:
‘Transfer-Encoding: chunked’,
Fix:
‘Transfer-Encoding: gzip,deflate’,

@laurynas-convertapi

@laurynas-convertapi, could you investigate and confirm, and if needed, apply the fix? Let me know If you need help from our internal developers.

@firepolo

As discussed using the chat with Paulius and Tomas a while back and since we are (still) seeying the same issues as metioned by @firepolo, hereby more details like you asked @tomasr78.

Lately we have been experiencing a lot of error 5001 error codes while trying to convert documents (docx, xlsx, png, jpg) to pdf. We are using the following function:

use ConvertApi\ConvertApi;
use ConvertApi\FileUpload;

$result = ConvertApi::convert('pdf', [
    'File' => new FileUpload($fileStream, $fileName),
    'StoreFile' => false,
    'Timeout' => 30,
]);

We thought we were delivering wrong/corrupt documents, since all API calls worked properly. However, we were unable to find the reason the error 5001 was triggered (unable to convert uploaded document). Even rewriting the above code, using files in stead of streams, got the same 5001 error. Contacting support, and uploading the documents manually resulted in the correct conversions.

After doing some research, we found the issue in the upload function, using the FileUpload class.

On macOS we have written a simple test script:

#!/opt/homebrew/opt/[email protected]/bin/php
<?php
require_once('./vendor/autoload.php');

use ConvertApi\ConvertApi;
use ConvertApi\Client;

ConvertApi::setApiSecret('your-api-secret');

$file = 'test.docx';

$fileContents = file_get_contents($file);
$crc = hash('crc32b', $fileContents);

echo "Original CRC: " . $crc . PHP_EOL;

$result = (new Client)->upload($file, $file);

echo "Upload url: " . $result['Url'] . PHP_EOL;

$crc = hash('crc32b', file_get_contents($result['Url']));
echo "Uploaded CRC: " . $crc . PHP_EOL;

To replicate, in an empty folder use composer require convertapi/convertapi-php, copy in the above test.php script (make it executable chmod +x test.php, also place the test.docx document (provided by the repository) in the same folder.

We get the following result running the above script:

➜  temp ./test.php
Original CRC: 7937aa29
Upload url: https://v2.convertapi.com/d/2a0r9vequ6efsm5vsa1xdzf33nahgbfc
Uploaded CRC: 6319cecd

The upload URL provides a corrupt file. Somewhere multiple bytes at the start of the file have been added, making it corrupt and therefore unable to convert.

Commenting out line #30 of /ConvertApi/Client solves the issue:

➜  temp ./test.php
Original CRC: 7937aa29
Upload url: https://v2.convertapi.com/d/sor8qc8zufhth05hjfpa459pexgpo51w
Uploaded CRC: 7937aa29

With line #30 of /ConvertApi/Client disabled, our original function worked properly again and the 5001 error disappeared.

Tested on:

PHP: 8.3.7 (attached a phpinfo() export)
macOS: 14.4.1 (23E224)

We do not believe this is an issue of the PHP Client. However, somewhere the uploaded file is getting altered/corrupt using chunked Transfer-Encoding.

Thanks in advance for looking in to this matter.

@orlu-nd

@orlu-nd Could you provide the original and corrupted files? Please post them here for debugging.

➜ temp ./test.php
Original CRC: 7937aa29
Upload url: https://v2.convertapi.com/d/zp4b0kc380zeyxlj0aq1l2phso5bd0l9
Uploaded CRC: 6319cecd

Original file: test.docx
Corrupted/altered file (from the Upload url): test(1).docx

@firepolo

Hello, could you try replacing upload function in file lib/ConvertApi/Client.php
just replace function and please reply with test results

` public function upload($file_or_resource, $fileName)
{
$ch = $this->initCurl('upload', ConvertApi::$uploadTimeout, []);

    if (!file_exists($file_or_resource))
    {
        throw new Error\File("Unable to open file {$file_or_resource}");
    }

    curl_setopt($ch, CURLOPT_POST, true);

    // Define the file to be uploaded
    $cfile = new \CURLFile($file_or_resource, mime_content_type($file_or_resource), basename($file_or_resource));

    // Attach the file to the POST fields
    $post_fields = array('file' => $cfile);
    curl_setopt($ch, CURLOPT_POSTFIELDS, $post_fields);

    // Set additional options
    curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
    curl_setopt($ch, CURLOPT_MAXREDIRS, 10);

    try
    {
        $result = $this->execute($ch);
    }
    catch (\Exception $e)
    {
        fclose($fp);
        throw $e;
    }

    return $result;
}`

As discussed using the chat with Paulius and Tomas a while back and since we are (still) seeying the same issues as metioned by @firepolo, hereby more details like you asked @tomasr78.

Lately we have been experiencing a lot of error 5001 error codes while trying to convert documents (docx, xlsx, png, jpg) to pdf. We are using the following function:
use ConvertApi\ConvertApi;
use ConvertApi\FileUpload;

$result = ConvertApi::convert('pdf', [
    'File' => new FileUpload($fileStream, $fileName),
    'StoreFile' => false,
    'Timeout' => 30,
]);
We thought we were delivering wrong/corrupt documents, since all API calls worked properly. However, we were unable to find the reason the error 5001 was triggered (unable to convert uploaded document). Even rewriting the above code, using files in stead of streams, got the same 5001 error. Contacting support, and uploading the documents manually resulted in the correct conversions.

After doing some research, we found the issue in the upload function, using the FileUpload class.

On macOS we have written a simple test script:
#!/opt/homebrew/opt/[email protected]/bin/php
<?php
require_once('./vendor/autoload.php');

use ConvertApi\ConvertApi;
use ConvertApi\Client;

ConvertApi::setApiSecret('your-api-secret');

$file = 'test.docx';

$fileContents = file_get_contents($file);
$crc = hash('crc32b', $fileContents);

echo "Original CRC: " . $crc . PHP_EOL;

$result = (new Client)->upload($file, $file);

echo "Upload url: " . $result['Url'] . PHP_EOL;

$crc = hash('crc32b', file_get_contents($result['Url']));
echo "Uploaded CRC: " . $crc . PHP_EOL;
To replicate, in an empty folder use composer require convertapi/convertapi-php, copy in the above test.php script (make it executable chmod +x test.php, also place the test.docx document (provided by the repository) in the same folder.

We get the following result running the above script:
➜  temp ./test.php
Original CRC: 7937aa29
Upload url: https://v2.convertapi.com/d/2a0r9vequ6efsm5vsa1xdzf33nahgbfc
Uploaded CRC: 6319cecd
The upload URL provides a corrupt file. Somewhere multiple bytes at the start of the file have been added, making it corrupt and therefore unable to convert.

Commenting out line #30 of /ConvertApi/Client solves the issue:
➜  temp ./test.php
Original CRC: 7937aa29
Upload url: https://v2.convertapi.com/d/sor8qc8zufhth05hjfpa459pexgpo51w
Uploaded CRC: 7937aa29
With line #30 of /ConvertApi/Client disabled, our original function worked properly again and the 5001 error disappeared.

Tested on:

PHP: 8.3.7 (attached a phpinfo() export) macOS: 14.4.1 (23E224)

We do not believe this is an issue of the PHP Client. However, somewhere the uploaded file is getting altered/corrupt using chunked Transfer-Encoding.

Thanks in advance for looking in to this matter.

@firepolo

I have done some tests.

PASSES: The uploaded file on your servers (and provided by the Uploaded URL) has the exact same CRC as uploaded.
FAILS: CRC differs from uploaded one.

Test 1: Your suggested code. PASSES

public function upload($file_or_resource, $fileName)
{	      
  $ch = $this->initCurl('upload', ConvertApi::$uploadTimeout, []);
  
  if (!file_exists($file_or_resource))
  {
	    throw new Error\File("Unable to open file {$file_or_resource}");
  }
  
  curl_setopt($ch, CURLOPT_POST, true);
  
  // Define the file to be uploaded
  $cfile = new \CURLFile($file_or_resource, mime_content_type($file_or_resource), basename($file_or_resource));
  
  // Attach the file to the POST fields
  $post_fields = array('file' => $cfile);
  curl_setopt($ch, CURLOPT_POSTFIELDS, $post_fields);
  
  // Set additional options
  curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
  curl_setopt($ch, CURLOPT_MAXREDIRS, 10);
  
  try
  {
	    $result = $this->execute($ch);
  }
  catch (\Exception $e)
  {
	    fclose($fp);
	    throw $e;
  }
  
  return $result;
}

Test 2: Your suggested code (With Headers from original code). FAILS

public function upload($file_or_resource, $fileName)
{	

  $headers = array_merge($this->defaultHeaders(), [
    'Content-Type: application/octet-stream',
    'Transfer-Encoding: chunked',
    "Content-Disposition: attachment; filename*=UTF-8''" . rawurlencode($fileName),
  ]);
      
  $ch = $this->initCurl('upload', ConvertApi::$uploadTimeout, $headers);
  
  if (!file_exists($file_or_resource))
  {
	    throw new Error\File("Unable to open file {$file_or_resource}");
  }
  
  curl_setopt($ch, CURLOPT_POST, true);
  
  // Define the file to be uploaded
  $cfile = new \CURLFile($file_or_resource, mime_content_type($file_or_resource), basename($file_or_resource));
  
  // Attach the file to the POST fields
  $post_fields = array('file' => $cfile);
  curl_setopt($ch, CURLOPT_POSTFIELDS, $post_fields);
  
  // Set additional options
  curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
  curl_setopt($ch, CURLOPT_MAXREDIRS, 10);
  
  try
  {
	    $result = $this->execute($ch);
  }
  catch (\Exception $e)
  {
	    fclose($fp);
	    throw $e;
  }
  
  return $result;
}

Test 3: Your suggested code (With Headers, commented out 'Transfer-Encoding'). FAILS

public function upload($file_or_resource, $fileName)
{	

  $headers = array_merge($this->defaultHeaders(), [
    'Content-Type: application/octet-stream',
    //'Transfer-Encoding: chunked',
    "Content-Disposition: attachment; filename*=UTF-8''" . rawurlencode($fileName),
  ]);
      
  $ch = $this->initCurl('upload', ConvertApi::$uploadTimeout, $headers);
  
  if (!file_exists($file_or_resource))
  {
	    throw new Error\File("Unable to open file {$file_or_resource}");
  }
  
  curl_setopt($ch, CURLOPT_POST, true);
  
  // Define the file to be uploaded
  $cfile = new \CURLFile($file_or_resource, mime_content_type($file_or_resource), basename($file_or_resource));
  
  // Attach the file to the POST fields
  $post_fields = array('file' => $cfile);
  curl_setopt($ch, CURLOPT_POSTFIELDS, $post_fields);
  
  // Set additional options
  curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
  curl_setopt($ch, CURLOPT_MAXREDIRS, 10);
  
  try
  {
	    $result = $this->execute($ch);
  }
  catch (\Exception $e)
  {
	    fclose($fp);
	    throw $e;
  }
  
  return $result;
}

Test 4: Your suggested code (With Headers, commented out 'Content-Type'). PASSES

public function upload($file_or_resource, $fileName)
{	

  $headers = array_merge($this->defaultHeaders(), [
    //'Content-Type: application/octet-stream',
    'Transfer-Encoding: chunked',
    "Content-Disposition: attachment; filename*=UTF-8''" . rawurlencode($fileName),
  ]);
      
  $ch = $this->initCurl('upload', ConvertApi::$uploadTimeout, $headers);
  
  if (!file_exists($file_or_resource))
  {
	    throw new Error\File("Unable to open file {$file_or_resource}");
  }
  
  curl_setopt($ch, CURLOPT_POST, true);
  
  // Define the file to be uploaded
  $cfile = new \CURLFile($file_or_resource, mime_content_type($file_or_resource), basename($file_or_resource));
  
  // Attach the file to the POST fields
  $post_fields = array('file' => $cfile);
  curl_setopt($ch, CURLOPT_POSTFIELDS, $post_fields);
  
  // Set additional options
  curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
  curl_setopt($ch, CURLOPT_MAXREDIRS, 10);
  
  try
  {
	    $result = $this->execute($ch);
  }
  catch (\Exception $e)
  {
	    fclose($fp);
	    throw $e;
  }
  
  return $result;
}

Test 5: Your suggested code (With Headers, commented out 'Content-Type' and 'Transfer-Encoding'). PASSES

public function upload($file_or_resource, $fileName)
{	

  $headers = array_merge($this->defaultHeaders(), [
    //'Content-Type: application/octet-stream',
    //'Transfer-Encoding: chunked',
    "Content-Disposition: attachment; filename*=UTF-8''" . rawurlencode($fileName),
  ]);
      
  $ch = $this->initCurl('upload', ConvertApi::$uploadTimeout, $headers);
  
  if (!file_exists($file_or_resource))
  {
	    throw new Error\File("Unable to open file {$file_or_resource}");
  }
  
  curl_setopt($ch, CURLOPT_POST, true);
  
  // Define the file to be uploaded
  $cfile = new \CURLFile($file_or_resource, mime_content_type($file_or_resource), basename($file_or_resource));
  
  // Attach the file to the POST fields
  $post_fields = array('file' => $cfile);
  curl_setopt($ch, CURLOPT_POSTFIELDS, $post_fields);
  
  // Set additional options
  curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
  curl_setopt($ch, CURLOPT_MAXREDIRS, 10);
  
  try
  {
	    $result = $this->execute($ch);
  }
  catch (\Exception $e)
  {
	    fclose($fp);
	    throw $e;
  }
  
  return $result;
}

Test 6: Orignal code. FAILS

public function upload($file_or_resource, $fileName)
{
    $headers = array_merge(
        $this->defaultHeaders(),
        [
            'Content-Type: application/octet-stream',
            'Transfer-Encoding: chunked',
            "Content-Disposition: attachment; filename*=UTF-8''" . rawurlencode($fileName),
        ]
    );

    $ch = $this->initCurl('upload', ConvertApi::$uploadTimeout, $headers);

    if (is_resource($file_or_resource)) {
        $fp = $file_or_resource;
    } else {
        $fp = fopen($file_or_resource, 'rb');
        
        curl_setopt($ch, CURLOPT_INFILESIZE, filesize($file_or_resource));
    }

    if (!$fp)
    {
        throw new Error\File("Unable to open file {$file_or_resource}");
    }

    curl_setopt($ch, CURLOPT_POST, true);
    curl_setopt($ch, CURLOPT_INFILE, $fp);            

    try
    {
        $result = $this->execute($ch);
    }
    catch (\Exception $e)
    {
        fclose($fp);
        throw $e;
    }

    fclose($fp);

    return $result;
}

Test 7: Orignal code (With Headers, commented out 'Transfer-Encoding'). FAILS

public function upload($file_or_resource, $fileName)
{
    $headers = array_merge(
        $this->defaultHeaders(),
        [
            'Content-Type: application/octet-stream',
            //'Transfer-Encoding: chunked',
            "Content-Disposition: attachment; filename*=UTF-8''" . rawurlencode($fileName),
        ]
    );

    $ch = $this->initCurl('upload', ConvertApi::$uploadTimeout, $headers);

    if (is_resource($file_or_resource)) {
        $fp = $file_or_resource;
    } else {
        $fp = fopen($file_or_resource, 'rb');
        
        curl_setopt($ch, CURLOPT_INFILESIZE, filesize($file_or_resource));
    }

    if (!$fp)
    {
        throw new Error\File("Unable to open file {$file_or_resource}");
    }

    curl_setopt($ch, CURLOPT_POST, true);
    curl_setopt($ch, CURLOPT_INFILE, $fp);            

    try
    {
        $result = $this->execute($ch);
    }
    catch (\Exception $e)
    {
        fclose($fp);
        throw $e;
    }

    fclose($fp);

    return $result;
}

Test 8: Orignal code (With Headers, commented out 'Content-Type'). FAILS

public function upload($file_or_resource, $fileName)
{
    $headers = array_merge(
        $this->defaultHeaders(),
        [
            //'Content-Type: application/octet-stream',
            'Transfer-Encoding: chunked',
            "Content-Disposition: attachment; filename*=UTF-8''" . rawurlencode($fileName),
        ]
    );

    $ch = $this->initCurl('upload', ConvertApi::$uploadTimeout, $headers);

    if (is_resource($file_or_resource)) {
        $fp = $file_or_resource;
    } else {
        $fp = fopen($file_or_resource, 'rb');
        
        curl_setopt($ch, CURLOPT_INFILESIZE, filesize($file_or_resource));
    }

    if (!$fp)
    {
        throw new Error\File("Unable to open file {$file_or_resource}");
    }

    curl_setopt($ch, CURLOPT_POST, true);
    curl_setopt($ch, CURLOPT_INFILE, $fp);            

    try
    {
        $result = $this->execute($ch);
    }
    catch (\Exception $e)
    {
        fclose($fp);
        throw $e;
    }

    fclose($fp);

    return $result;
}

Test 9: Orignal code (With Headers, commented out 'Content-Type' AND 'Transfer-Encoding'). PASSES

public function upload($file_or_resource, $fileName)
{
    $headers = array_merge(
        $this->defaultHeaders(),
        [
            //'Content-Type: application/octet-stream',
            //'Transfer-Encoding: chunked',
            "Content-Disposition: attachment; filename*=UTF-8''" . rawurlencode($fileName),
        ]
    );

    $ch = $this->initCurl('upload', ConvertApi::$uploadTimeout, $headers);

    if (is_resource($file_or_resource)) {
        $fp = $file_or_resource;
    } else {
        $fp = fopen($file_or_resource, 'rb');
        
        curl_setopt($ch, CURLOPT_INFILESIZE, filesize($file_or_resource));
    }

    if (!$fp)
    {
        throw new Error\File("Unable to open file {$file_or_resource}");
    }

    curl_setopt($ch, CURLOPT_POST, true);
    curl_setopt($ch, CURLOPT_INFILE, $fp);            

    try
    {
        $result = $this->execute($ch);
    }
    catch (\Exception $e)
    {
        fclose($fp);
        throw $e;
    }

    fclose($fp);

    return $result;
}

Test 10: Orignal code (With Headers, commented out 'Content-Type' AND 'Transfer-Encoding' as per @firepolo suggestion). PASSES

public function upload($file_or_resource, $fileName)
{
    $headers = array_merge(
        $this->defaultHeaders(),
        [
            //'Content-Type: application/octet-stream',
            'Transfer-Encoding: gzip,deflate',
            "Content-Disposition: attachment; filename*=UTF-8''" . rawurlencode($fileName),
        ]
    );

    $ch = $this->initCurl('upload', ConvertApi::$uploadTimeout, $headers);

    if (is_resource($file_or_resource)) {
        $fp = $file_or_resource;
    } else {
        $fp = fopen($file_or_resource, 'rb');
        
        curl_setopt($ch, CURLOPT_INFILESIZE, filesize($file_or_resource));
    }

    if (!$fp)
    {
        throw new Error\File("Unable to open file {$file_or_resource}");
    }

    curl_setopt($ch, CURLOPT_POST, true);
    curl_setopt($ch, CURLOPT_INFILE, $fp);            

    try
    {
        $result = $this->execute($ch);
    }
    catch (\Exception $e)
    {
        fclose($fp);
        throw $e;
    }

    fclose($fp);

    return $result;
}

Test 11: Orignal code (With Headers, 'Transfer-Encoding' as per @firepolo suggestion). PASSES

public function upload($file_or_resource, $fileName)
{
    $headers = array_merge(
        $this->defaultHeaders(),
        [
            'Content-Type: application/octet-stream',
            'Transfer-Encoding: gzip,deflate',
            "Content-Disposition: attachment; filename*=UTF-8''" . rawurlencode($fileName),
        ]
    );

    $ch = $this->initCurl('upload', ConvertApi::$uploadTimeout, $headers);

    if (is_resource($file_or_resource)) {
        $fp = $file_or_resource;
    } else {
        $fp = fopen($file_or_resource, 'rb');
        
        curl_setopt($ch, CURLOPT_INFILESIZE, filesize($file_or_resource));
    }

    if (!$fp)
    {
        throw new Error\File("Unable to open file {$file_or_resource}");
    }

    curl_setopt($ch, CURLOPT_POST, true);
    curl_setopt($ch, CURLOPT_INFILE, $fp);            

    try
    {
        $result = $this->execute($ch);
    }
    catch (\Exception $e)
    {
        fclose($fp);
        throw $e;
    }

    fclose($fp);

    return $result;
}

Couldn't reproduce the issue on my machine (Ubuntu Linux + Docker), probably depends on environment. Removed content type and transfer encoding headers from file upload. Released client version 2.0.1.

@laurynas-convertapi

@laurynas-convertapi Client approved that your fix is working fine. Good job.

Navigation Menu

Repository navigation

Conversion error #56

Test 1: Your suggested code. PASSES

Test 2: Your suggested code (With Headers from original code). FAILS

Test 3: Your suggested code (With Headers, commented out 'Transfer-Encoding'). FAILS

Test 4: Your suggested code (With Headers, commented out 'Content-Type'). PASSES

Test 5: Your suggested code (With Headers, commented out 'Content-Type' and 'Transfer-Encoding'). PASSES

Test 6: Orignal code. FAILS

Test 7: Orignal code (With Headers, commented out 'Transfer-Encoding'). FAILS

Test 8: Orignal code (With Headers, commented out 'Content-Type'). FAILS

Test 9: Orignal code (With Headers, commented out 'Content-Type' AND 'Transfer-Encoding'). PASSES

Test 10: Orignal code (With Headers, commented out 'Content-Type' AND 'Transfer-Encoding' as per @firepolo suggestion). PASSES

Test 11: Orignal code (With Headers, 'Transfer-Encoding' as per @firepolo suggestion). PASSES

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Conversion error #56

Description

Activity

tomasr78 commented on Jun 12, 2024

orlu-nd commented on Jun 21, 2024

tomasr78 commented on Jun 21, 2024

orlu-nd commented on Jun 21, 2024

gwcms commented on Jun 21, 2024

orlu-nd commented on Jun 21, 2024

Test 1: Your suggested code. PASSES

Test 2: Your suggested code (With Headers from original code). FAILS

Test 3: Your suggested code (With Headers, commented out 'Transfer-Encoding'). FAILS

Test 4: Your suggested code (With Headers, commented out 'Content-Type'). PASSES

Test 5: Your suggested code (With Headers, commented out 'Content-Type' and 'Transfer-Encoding'). PASSES

Test 6: Orignal code. FAILS

Test 7: Orignal code (With Headers, commented out 'Transfer-Encoding'). FAILS

Test 8: Orignal code (With Headers, commented out 'Content-Type'). FAILS

Test 9: Orignal code (With Headers, commented out 'Content-Type' AND 'Transfer-Encoding'). PASSES

Test 10: Orignal code (With Headers, commented out 'Content-Type' AND 'Transfer-Encoding' as per @firepolo suggestion). PASSES

Test 11: Orignal code (With Headers, 'Transfer-Encoding' as per @firepolo suggestion). PASSES

laurynas-convertapi commented on Jun 24, 2024

paulius-petkus commented on Jun 27, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions