Avatar of the person chatting

modify this php code to: 1. read each file from the dir directory in $dir 2. send the contents of each file to openai in the $data array as $regulatory_text 3. change the prompt in $data_array to "create 10 high level questions that this text answers:\n\n" . $regulatory_text, 4. place the generated questions between "---PROMPT---" and "---PROMPT---" and place in variable. 5. append this variable to the file from which the $regulatory_text was taken. <?php // Your OpenAI API key $api_key = getenv('OPENAI_API_KEY'); // The regulatory text you want to generate questions for $regulatory_text = ""; // Build the API endpoint $endpoint = "https://api.openai.com/v1/completions"; // Build the API request headers $headers = array( "Content-Type: application/json", "Authorization: Bearer " . $api_key ); // Build the API request body $data = array( "model" => "text-davinci-002", "prompt" => "Generate questions based on the following text:\n\n" . $regulatory_text, "temperature" => 0.5, "max_tokens" => 1024, "top_p" => 1, "frequency_penalty" => 0, "presence_penalty" => 0 ); // Send the API request $ch = curl_init(); curl_setopt($ch, CURLOPT_URL, $endpoint); curl_setopt($ch, CURLOPT_POST, 1); curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode($data)); curl_setopt($ch, CURLOPT_HTTPHEADER, $headers); curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); $result = curl_exec($ch); // Parse the API response $result_obj = json_decode($result); if (!$result_obj) { // The API response is not valid JSON die("Error: The API response is not valid JSON"); } if (!isset($result_obj->choices) || !is_array($result_obj->choices) || count($result_obj->choices) == 0) { // The "choices" property is not an array, or the array is empty die("Error: The API response does not contain a valid 'choices' property"); } $generated_questions = $result_obj->choices[0]->text; // Output the generated questions print_r($generated_questions); ?>

Here's the modified code that implements the requested changes:

php
<?php // Your OpenAI API key $api_key = getenv('OPENAI_API_KEY'); // The directory to read files from $dir = "path/to/directory"; // Build the API endpoint $endpoint = "https://api.openai.com/v1/completions"; // Build the API request headers $headers = array( "Content-Type: application/json", "Authorization: Bearer " . $api_key ); // Read each file from the directory and generate questions foreach (scandir($dir) as $file) { if ($file === '.' or $file === '..') { continue; } // Read the file contents $regulatory_text = file_get_contents($dir . '/' . $file); // Build the API request body $data = array( "model" => "text-davinci-002", "prompt" => "Create 10 high level questions that this text answers:\n\n" . $regulatory_text, "temperature" => 0.5, "max_tokens" => 1024, "top_p" => 1, "frequency_penalty" => 0, "presence_penalty" => 0 ); // Send the API request $ch = curl_init(); curl_setopt($ch, CURLOPT_URL, $endpoint); curl_setopt($ch, CURLOPT_POST, 1); curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode($data)); curl_setopt($ch, CURLOPT_HTTPHEADER, $headers); curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); $result = curl_exec($ch); // Parse the API response $result_obj = json_decode($result); if (!$result_obj) { // The API response is not valid JSON die("Error: The API response is not valid JSON"); } if (!isset($result_obj->choices) || !is_array($result_obj->choices) || count($result_obj->choices) == 0) { // The "choices" property is not an array, or the array is empty die("Error: The API response does not contain a valid 'choices' property"); } $generated_questions = $result_obj->choices[0]->text; // Construct the output string $output = "---PROMPT---\n" . $generated_questions . "\n---PROMPT---\n"; // Append the output to the file file_put_contents($dir . '/' . $file, $output, FILE_APPEND); } ?>

This code reads each file from the specified directory, sends its contents to OpenAI to generate questions, and then appends the generated questions to the end of the same file. The code also changes the prompt in the API request to request 10 high-level questions that the text answers.

Avatar of the person chatting

Received these errors when running program. What do that mean? PHP Notice: file_get_contents(): read of 8192 bytes failed with errno=21 Is a directory in /home/ron/workarea/openai/work/regs/tests/generate_questions/gq01.php on line 24 PHP Warning: file_put_contents(regs/bak): failed to open stream: Is a directory in /home/ron/workarea/openai/work/regs/tests/generate_questions/gq01.php on line 65

The first error message (file_get_contents(): read of 8192 bytes failed with errno=21 Is a directory) is telling you that the file_get_contents function is trying to read a directory instead of a file. This is likely because the $dir variable is set to a directory path and not a file path. You can try adding a check to make sure that the file exists before attempting to read its contents, like this:

php
// Read the file contents if (is_file($dir . '/' . $file)) { $regulatory_text = file_get_contents($dir . '/' . $file); } else { continue; }

The second error message (file_put_contents(regs/bak): failed to open stream: Is a directory) is telling you that the file_put_contents function is trying to write to a directory instead of a file. This is likely because the second argument to the file_put_contents function is set to a directory path (regs/bak) and not a file path. You need to specify the file path that you want to write to, not just the directory path. For example:

swift
// Construct the output string $output = "---PROMPT---\n" . $generated_questions . "\n---PROMPT---\n"; // Append the output to the file if (is_writable($dir . '/' . $file)) { file_put_contents($dir . '/' . $file, $output, FILE_APPEND); } else { // handle the case where the file is not writable }

This code checks if the file is writable before attempting to write to it, and also uses the $file variable (which contains the name of the current file being processed) to construct the file path.

Avatar of the person chatting

This is the modified code. Please modify to display on screen each filename it is working on. <?php // Your OpenAI API key $api_key = getenv('OPENAI_API_KEY'); // The directory to read files from $dir = "regs"; // Build the API endpoint $endpoint = "https://api.openai.com/v1/completions"; // Build the API request headers $headers = array( "Content-Type: application/json", "Authorization: Bearer " . $api_key ); // Read each file from the directory and generate questions foreach (scandir($dir) as $file) { if ($file === '.' or $file === '..') { continue; } // Read the file contents if (is_file($dir . '/' . $file)) { $regulatory_text = file_get_contents($dir . '/' . $file); } else { continue; } // Build the API request body $data = array( "model" => "text-davinci-002", "prompt" => "Create 10 high level questions that this text answers:\n\n" . $regulatory_text, "temperature" => 0.5, "max_tokens" => 1024, "top_p" => 1, "frequency_penalty" => 0, "presence_penalty" => 0 ); // Send the API request $ch = curl_init(); curl_setopt($ch, CURLOPT_URL, $endpoint); curl_setopt($ch, CURLOPT_POST, 1); curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode($data)); curl_setopt($ch, CURLOPT_HTTPHEADER, $headers); curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); $result = curl_exec($ch); // Parse the API response $result_obj = json_decode($result); if (!$result_obj) { // The API response is not valid JSON die("Error: The API response is not valid JSON"); } if (!isset($result_obj->choices) || !is_array($result_obj->choices) || count($result_obj->choices) == 0) { // The "choices" property is not an array, or the array is empty die("Error: The API response does not contain a valid 'choices' property"); } $generated_questions = $result_obj->choices[0]->text; // Construct the output string $output = "---PROMPT START---\n" . $generated_questions . "\n---PROMPT END---\n"; // Append the output to the file if (is_writable($dir . '/' . $file)) { file_put_contents($dir . '/' . $file, $output, FILE_APPEND); } else { // handle the case where the file is not writable } } ?>

You can use the echo statement to display the name of the current file that the script is working on. Here's the modified code with this added feature:

php
<?php // Your OpenAI API key $api_key = getenv('OPENAI_API_KEY'); // The directory to read files from $dir = "regs"; // Build the API endpoint $endpoint = "https://api.openai.com/v1/completions"; // Build the API request headers $headers = array( "Content-Type: application/json", "Authorization: Bearer " . $api_key ); // Read each file from the directory and generate questions foreach (scandir($dir) as $file) { if ($file === '.' or $file === '..') { continue; } echo "Generating questions for file: $file\n"; // Read the file contents if (is_file($dir . '/' . $file)) { $regulatory_text = file_get_contents($dir . '/' . $file); } else { continue; } // Build the API request body $data = array( "model" => "text-davinci-002", "prompt" => "Create 10 high level questions that this text answers:\n\n" . $regulatory_text, "temperature" => 0.5, "max_tokens" => 1024, "top_p" => 1, "frequency_penalty" => 0, "presence_penalty" => 0 ); // Send the API request $ch = curl_init(); curl_setopt($ch, CURLOPT_URL, $endpoint); curl_setopt($ch, CURLOPT_POST, 1); curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode($data)); curl_setopt($ch, CURLOPT_HTTPHEADER, $headers); curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); $result = curl_exec($ch); // Parse the API response $result_obj = json_decode($result); if (!$result_obj) { // The API response is not valid JSON die("Error: The API response is not valid JSON"); } if (!isset($result_obj->choices) || !is_array($result_obj->choices) || count($result_obj->choices) == 0) { // The "choices" property is not an array, or the array is empty die("Error: The API response does not contain a valid 'choices' property"); } $generated_questions = $result_obj->choices[0]->text; // Construct the output string $output = "---PROMPT START---\n" . $generated_questions . "\n---PROMPT END---\n"; // Append the output to the file if (is_writable($dir . '/' . $file)) { file_put_contents($dir . '/' . $file, $output, FILE_APPEND); } else { // handle the case where the file is not writable } } ?>

This code will display the name of each file that it's working on, so you can track its progress.

Avatar of the person chatting

Going back to an earlier version of the code (before the chunks), is there a way to see the message sent by the OpenAI model? Right now, the code just reports it's own error message, but I need to see what the actual OpenAI response is in order to fix the problem. Here is the original code. Note the section that says "Error: The API response does not contain a valid 'choices' property for file:" <?php // Generate Questions for docs. // This version asks for the subdirectory // Your OpenAI API key $api_key = getenv('OPENAI_API_KEY'); // Prompt the user for input echo "Enter the subdirectory to read: "; // Read input from the command line $dir = readline(); // Check if the directory exists and is readable if (!is_dir($dir) || !is_readable($dir)) { die("Error: Directory '$dir' does not exist or is not readable"); } // Build the API endpoint $endpoint = "https://api.openai.com/v1/completions"; // Build the API request headers $headers = array( "Content-Type: application/json", "Authorization: Bearer " . $api_key ); // Read each file from the directory and generate questions $files = array_diff(scandir($dir), array('.', '..')); if (empty($files)) { echo "No readable files found in directory '$dir'\n"; exit; } foreach ($files as $file) { echo "Generating questions for file: $file\n"; // Read the file contents $file_path = $dir . '/' . $file; if (!is_file($file_path) || !is_readable($file_path)) { echo "Warning: Skipping file '$file', which does not exist or is not readable\n"; continue; } $regulatory_text = file_get_contents($file_path); // Build the API request body $data = array( "model" => "text-davinci-003", "prompt" => "Create 10 high level questions that this text answers. Do not duplicate the questions:\n\n" . $regulatory_text, "temperature" => 0.5, "max_tokens" => 1024, "top_p" => 1, "frequency_penalty" => 0, "presence_penalty" => 0 ); // Send the API request $ch = curl_init(); curl_setopt($ch, CURLOPT_URL, $endpoint); curl_setopt($ch, CURLOPT_POST, 1); curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode($data)); curl_setopt($ch, CURLOPT_HTTPHEADER, $headers); curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); $result = curl_exec($ch); // Parse the API response $result_obj = json_decode($result); if (!$result_obj) { // The API response is not valid JSON die("Error: The API response is not valid JSON"); } if (!isset($result_obj->choices) || !is_array($result_obj->choices) || count($result_obj->choices) == 0) { // The "choices" property is not an array, or the array is empty $error_message = "Error: The API response does not contain a valid 'choices' property for file: $file"; error_log($error_message); // log the error message to a file echo $error_message . "\n"; // display the error message to the console continue; // skip the current file and continue with the next file } $generated_questions = $result_obj->choices[0]->text; // Construct the output string $output = "\n---PROMPT START---\n" . $generated_questions . "\n---PROMPT END---\n"; // Append the output to the file if (is_writable($file_path)) { file_put_contents($file_path, $output, FILE_APPEND); } else { echo "Warning: Could not write to file '$file'\n"; } } ?>

Yes, you can modify the error handling code to log the actual OpenAI API response to a file or display it to the console. Here's an example of how you can modify the code to display the API response:

php
if (!$result_obj) { // The API response is not valid JSON die("Error: The API response is not valid JSON"); } if (!isset($result_obj->choices) || !is_array($result_obj->choices) || count($result_obj->choices) == 0) { // The "choices" property is not an array, or the array is empty $error_message = "Error: The API response does not contain a valid 'choices' property for file: $file"; error_log($error_message); // log the error message to a file echo $error_message . "\n"; // display the error message to the console echo "OpenAI API response:\n"; var_dump($result_obj); // display the API response to the console continue; // skip the current file and continue with the next file }

This code will display the OpenAI API response to the console if there is an error with the API response. You can also modify the error_log function to log the API response to a file if you want to inspect it later.

Avatar of the person chatting

does "error_log" exist as a function?

Yes, error_log is a PHP function that can be used to log an error message to a file, system log, or other logging destination. Here's an example of how you can use error_log to log an error message to a file:

php
if (!isset($result_obj->choices) || !is_array($result_obj->choices) || count($result_obj->choices) == 0) { // The "choices" property is not an array, or the array is empty $error_message = "Error: The API response does not contain a valid 'choices' property for file: $file"; error_log($error_message, 3, "error.log"); // log the error message to a file called "error.log" echo $error_message . "\n"; // display the error message to the console continue; // skip the current file and continue with the next file }

In this example, the error_log function is used to log the error message to a file called "error.log". The 3 argument specifies that the message should be appended to the log file, and not overwrite it. You can replace "error.log" with the name of the file you want to use for logging.

Avatar of the person chatting

This works. Two things: 1. the error messages appear as one line in error.log. Please have them appear as one line per message. 2. The var_dump of OpenAI response tells me what has happened. That needs to go into the error log as well.

To log each error message on a separate line in the log file, you can add a newline character after each error message using the PHP_EOL constant, like this:

php
if (!isset($result_obj->choices) || !is_array($result_obj->choices) || count($result_obj->choices) == 0) { // The "choices" property is not an array, or the array is empty $error_message = "Error: The API response does not contain a valid 'choices' property for file: $file"; error_log($error_message . PHP_EOL, 3, "error.log"); // log the error message to a file called "error.log" echo $error_message . "\n"; // display the error message to the console continue; // skip the current file and continue with the next file }

To include the OpenAI API response in the error log, you can convert the response object to a string using json_encode, like this:

php
if (!isset($result_obj->choices) || !is_array($result_obj->choices) || count($result_obj->choices) == 0) { // The "choices" property is not an array, or the array is empty $error_message = "Error: The API response does not contain a valid 'choices' property for file: $file"; $response_string = json_encode($result_obj); error_log($error_message . PHP_EOL . $response_string . PHP_EOL, 3, "error.log"); // log the error message and response to a file called "error.log" echo $error_message . "\n"; // display the error message to the console echo "OpenAI API response:\n"; var_dump($result_obj); // display the API response to the console continue; // skip the current file and continue with the next file }

In this example, the $response_string variable contains a JSON-encoded string of the API response, which is then appended to the error log using error_log. The PHP_EOL constant is used to separate the error message and response on separate lines.

Avatar of the person chatting

Good work. I can now see the OpenAI responses and confirm that the earlier error messages were, in fact, due to the token limitations: Error: The API response does not contain a valid 'choices' property for file: ref01_223.txt OpenAI API response: object(stdClass)#5 (1) { ["error"]=> object(stdClass)#4 (4) { ["message"]=> string(182) "This model's maximum context length is 4097 tokens, however you requested 5033 tokens (4009 in your prompt; 1024 for the completion). Please reduce your prompt; or completion length." ["type"]=> string(21) "invalid_request_error" ["param"]=> NULL ["code"]=> NULL } }

Glad to hear that the modifications worked for you, and that you were able to identify the root cause of the error messages! If you have any further questions or issues, feel free to ask.