Replicate ChatGPT Price Forecasting

Replicate ChatGPT Price Forecasting#

This notebook gives a concise, reader-friendly tour of the project workflow and the core outputs used in the replication.

Summary#

The Lopez-Lira and Tang (2023) paper ā€œCan ChatGPT Forecast Stock Price Movements? Return Predictability and Large Language Modelsā€ documents the capability of LLMs, such as ChatGPT, to predict stock market reactions from news headlines without direct financial training. The papers results suggest forecasting ability generally increases with model size and strategy returns decline as LLM usage within the finanical domain increases. We aim to replicate their results, specifically looking at strategy hit rates, and portfolio returns results.

Methodology at a glance#

  • Data foundation: pull CRSP prices and RavenPack headlines from WRDS.

  • Data cleaning: map entities to tickers, remove low-quality/duplicate signals, and align headline timing to trading logic.

  • NLP labeling: submit batched headlines to OpenAI and parse labels into structured sentiment scores.

  • Portfolio construction: convert firm-day sentiment into long/short return series under multiple sample restrictions.

  • Reporting: generate summary tables and figures used in the write-up.

Pipeline context for this notebook#

The data shown below comes from the doit pipeline in dodo.py.

  • pull:crsp_stock runs pull_CRSP_stock.py and writes CRSP_stock_daily.parquet and CRSP_unique_tickers.parquet.

  • pull:ravenpack runs pull_ravenpack.py and writes RAVENPACK.parquet.

  • clean_data:clean_ravenpack runs clean_ravenpack.py and writes RAVENPACK_cleaned.parquet.

  • process:generate_batched_requests, process:submit_headlines_to_openai, and process:create_firm_day_score produce daily_headline_polarity.parquet.

  • Downstream scripts create_portfolios.py and create_table1.py generate return panels and final table CSVs used in this guide.

Step 0. Setup and Key Paths#

Start by confirming which pipeline artifacts exist before loading data. This prevents downstream errors and makes the notebook reproducible across machines and collaborators.

The code imports project settings from settings.py, builds the canonical file map, and prints an existence table so we can verify which stages of the pipeline have already been run.

import json
from pathlib import Path
import pandas as pd

from notebook_helper import (
    generate_single_request_jsonl,
    get_rp_timing_stats,
    printable_table,
)
from settings import config

MANUAL_DATA_DIR = Path(config("MANUAL_DATA_DIR"))
DATA_DIR = Path(config("DATA_DIR"))
OUTPUT_DIR = Path(config("OUTPUT_DIR"))

paths = {
    "CRSP Stock Data": DATA_DIR / "CRSP_stock_daily.parquet",
    "RavenPack Full": DATA_DIR / "RAVENPACK.parquet",
    "RavenPack Clean": DATA_DIR / "RAVENPACK_cleaned.parquet",
    "Daily Headline Scores": DATA_DIR / "daily_headline_polarity.parquet",
    "Portfolio Returns": DATA_DIR / "portfolio_daily_returns.parquet",
    "Table1 (Oct 2021 - May 2024)": OUTPUT_DIR / "table1_overnight_paper_sample.csv",
    "Table1 (Oct 2021 - March 2026)": OUTPUT_DIR / "table1_overnight_full_sample.csv",
    "Paper Table Data": MANUAL_DATA_DIR / "paper_table1.csv",
}

pd.DataFrame(
    {
        "file": list(paths.keys()),
        "exists": [p.exists() for p in paths.values()],
    }
)
file exists
0 CRSP Stock Data True
1 RavenPack Full True
2 RavenPack Clean True
3 Daily Headline Scores True
4 Portfolio Returns True
5 Table1 (Oct 2021 - May 2024) True
6 Table1 (Oct 2021 - March 2026) True
7 Paper Table Data True

Step 1. Load Cleaned RavenPack and CRSP#

In order to replicate the results in this paper, we need market data from CRSP and news data from RavenPack. We are ignoring intraday data. Additionally, we use the RPA Entity Mapping File (wrds_rpa_company_mappings) in order to facilitate the mapping of RavenPack entities to company tickers.

We pulled data between 2021-10-1 and 2026-03-01 for this project. Our replication is concerned with the paperā€™s timebounds of 2021-10-01 to 2024-05-31, and additional analysis was completed on the full sample.

  • The CRSP dataset contains stock market data, including prices, returns, and shares outstanding, for securities traded on major U.S. exchanges. This dataset is essential for measuring market performance and calculating variables such as market equity.

  • The RavenPack dataset, using over 40,000 sources, provides real-time news analytics, including sentiment analysis and event data focused on business and financial applications. Data includes news and social media content, allowing for comprehensive analysis of financial markets.

  • The RPA Entity Mapping File provides a variety of security identifiers (ex. ISINs, CUSIPs, etc.) to allow you to identify the 90,000+ companies.

Pipeline step that generated this data

  • CRSP_stock_daily.parquet is generated by pull:crsp_stock via pull_CRSP_stock.py.

  • RAVENPACK_cleaned.parquet is generated by clean_data:clean_ravenpack via clean_ravenpack.py (after pull:ravenpack).

Our pulled CRSP dataset is limited to the world of stocks specified in the paper:

WHERE
    (
        primaryexch IN ( 'N', 'A', 'Q' ) AND
        conditionaltype = 'RW' AND
        tradingstatusflg = 'A' AND
        dlycaldt >= '{start_date}' AND
        dlycaldt <= '{end_date}'
        {permno_filter}
    ) ;

The clean_ravenpack process was designed to replicate the data cleaning procedure described in the paper. This includes filtering RavenPack data to match CRSP tickers, ensure a baseline relevancy rating of 0.60, deduplicating firm-day headlines using Optimal String Alignment (OSA) similarity, and aligning headline timestamps to Eastern Time. Intraday headlines are excluded, and overnight headlines are adjusted to ensure proper alignment with trading day per teh overnight cutoff at 4:00PM ET. This step ensures that the cleaned dataset is consistent with the methodology outlined in the paper.

rp = pd.read_parquet(paths["RavenPack Clean"]) if paths["RavenPack Clean"].exists() else pd.DataFrame()
crsp = pd.read_parquet(paths["CRSP Stock Data"]) if paths["CRSP Stock Data"].exists() else pd.DataFrame()

print("RavenPack cleaned shape:", rp.shape)
print("CRSP Stock Data shape:", crsp.shape)

display(rp.head(3))
display(crsp.head(3))
del crsp

RavenPack cleaned shape: (209714, 8)
CRSP Stock Data shape: (7467730, 8)
rp_entity_id map_ticker entity_name timestamp_utc rpa_date_utc timestamp_et headline_date headline
0 00067A HUM Humana Inc. 2025-01-09 21:30:00.623 2025-01-09 2025-01-09 16:30:00.623000-05:00 2025-01-10 Humana Inc. to Release Fourth Quarter 2024 Res...
1 00067A HUM Humana Inc. 2025-11-12 13:45:00.460 2025-11-12 2025-11-12 08:45:00.460000-05:00 2025-11-12 ProgenyHealth Announces Collaboration with Hum...
2 00067A HUM Humana Inc. 2025-07-31 11:00:04.767 2025-07-31 2025-07-31 07:00:04.767000-04:00 2025-07-31 Exact Sciences and Humana Expand Colorectal Ca...
permno permco ticker primaryexch date dlycap dlyopen dlyclose
0 10026 7976 JJSF Q 2021-10-01 2932065.76 153.85 153.64
1 10028 7978 ELA A 2021-10-01 109584.75 4.11 4.07
2 10032 7980 PLXS Q 2021-10-01 2554520.76 89.59 91.08

Headline Timing Diagnostics#

stats, table = get_rp_timing_stats(rp)

display(stats.style)
display(table.style)

Intraday rows remaining (expected ~0): 0
  n_rows n_tickers min_ts max_ts min_headline_date max_headline_date
0 209714 5505 2021-09-30 20:21:21.593000-04:00 2026-02-28 18:00:03.460000-05:00 2021-10-01 2026-03-01
  sample rows total headline dates rolled over
0 Oct 2021 - May 2024 130111 23798
1 Oct 2021 - Feb 2026 209714 38707

Step 3. Generate Batched Requests (Single-Row Example)#

Pipeline step that generated this data

  • openai_headline_requests.*.jsonl files are generated by process:generate_batched_requests via generate_batched_requests.py.

  • id_to_row_mapping.*.json files are generated by process:generate_batched_requests via generate_batched_requests.py.

This step demonstrates the generate_batched_requests.py logic on a single RavenPack row to replicate the pipeline behavior.

In order to retrieve OpenAI responses to our headline prompts, we must create batch files for asynchornous submission. The RavenPack data is broken up into chunks (40,000 being the default) and each request is a single line of json (referred to as JSONL). We also create a separate mapping file for ease of processing later on.

jsonl_content, mapping_content = generate_single_request_jsonl(rp.head(1).copy())

Wrote requests jsonl: /tmp/tmproknete3/openai_headline_requests.single.jsonl
Number of headlines batched: 1
Wrote id to row mapping json: /tmp/tmproknete3/id_to_row_mapping.single.json

print("JSONL content inline:")
display(jsonl_content)

JSONL content inline:

'{"custom_id": "rp-0", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gpt-3.5-turbo", "temperature": 0, "messages": [{"role": "system", "content": "Forget all your previous instructions. Pretend you are a financial expert. You are a financial expert with stock recommendation experience. Answer \\"YES\\" if good news, \\"NO\\" if bad news, or \\"UNKNOWN\\" if uncertain in the first line. Then elaborate with one short and concise sentence on the next line."}, {"role": "user", "content": "Is this headline good or bad for the stock price of Humana Inc. in the short term?\\nHeadline: Humana Inc. to Release Fourth Quarter 2024 Results on February 11, 2025"}]}}\n'

print("JSONL content formatted:")
print(
    json.dumps(
        json.loads(jsonl_content), sort_keys=True, indent=2, separators=(",", ": ")
    )
)

JSONL content formatted:
{
  "body": {
    "messages": [
      {
        "content": "Forget all your previous instructions. Pretend you are a financial expert. You are a financial expert with stock recommendation experience. Answer \"YES\" if good news, \"NO\" if bad news, or \"UNKNOWN\" if uncertain in the first line. Then elaborate with one short and concise sentence on the next line.",
        "role": "system"
      },
      {
        "content": "Is this headline good or bad for the stock price of Humana Inc. in the short term?\nHeadline: Humana Inc. to Release Fourth Quarter 2024 Results on February 11, 2025",
        "role": "user"
      }
    ],
    "model": "gpt-3.5-turbo",
    "temperature": 0
  },
  "custom_id": "rp-0",
  "method": "POST",
  "url": "/v1/chat/completions"
}

print("ID to Row Mapping:")
print(
    json.dumps(
        json.loads(mapping_content), sort_keys=True, indent=2, separators=(",", ": ")
    )
)

ID to Row Mapping:
{
  "rp-0": {
    "date": "2025-01-10",
    "entity_name": "Humana Inc.",
    "ticker": "HUM"
  }
}

Step 4. Submit Headlines to OpenAI Batch API#

Pipeline step that generated this data

  • openai_headline_batch_output.*.jsonl files are generated by process:submit_headlines_to_openai via submit_headlines_to_openai.py.

  • openai_headline_batch_metadata.*.json files are generated by process:submit_headlines_to_openai via submit_headlines_to_openai.py.

All batch request files matching openai_headline_requests.*.jsonl are then submitted to OpenAI via the process:submit_headlines_to_openai step, with an SLA of 24 hours for a response. The pipeline blocks while waiting for a completion status for all submitted files. Results are written to the DATA_DIR as openai_headline_batch_output.*.jsonl with corresponding metadata files openai_headline_batch_metadata.*.json.

We prompt the LLM for one of three responses

  • YES: headline was deemed positive for stock price forecast

  • NO: headline was deemed negative for stock price forecast

  • UNKNOWN: headline was deemed neutral for stock price forecast

Below is a success response from OpenAI.

data = pd.read_json(DATA_DIR / "openai_headline_batch_output.1.jsonl", lines=True).head(1)
display(data.iloc[0]['response'])

{'status_code': 200,
 'request_id': '20c04669-5262-44e6-8fca-cc48e5b5d47b',
 'body': {'id': 'chatcmpl-DJUCVBfr6XgXN8m415PXlAhTb9NaR',
  'object': 'chat.completion',
  'created': 1773536695,
  'model': 'gpt-3.5-turbo-0125',
  'choices': [{'index': 0,
    'message': {'role': 'assistant',
     'content': 'UNKNOWN\nThe stock price may fluctuate depending on the content of the results and market expectations.',
     'refusal': None,
     'annotations': []},
    'logprobs': None,
    'finish_reason': 'stop'}],
  'usage': {'prompt_tokens': 117,
   'completion_tokens': 19,
   'total_tokens': 136,
   'prompt_tokens_details': {'cached_tokens': 0, 'audio_tokens': 0},
   'completion_tokens_details': {'reasoning_tokens': 0,
    'audio_tokens': 0,
    'accepted_prediction_tokens': 0,
    'rejected_prediction_tokens': 0}},
  'service_tier': 'default',
  'system_fingerprint': None}}

Step 5. Firm-Day Sentiment Output#

The next step in our pipeline processes the OpenAI responses, and aggregates the modelā€™s rating of each headline.

Pipeline step that generated this data

  • daily_headline_polarity.parquet is generated by process:create_firm_day_score in create_firmday_score.py

We parse each response, and if the answer for a headline was YES it gets a score of 1, NO gets a score of -1, and UNKNOWN gets a score of 0. We group the data set by date and ticker, and aggregate the total headlines, total score sum, and overall polarity (positive or negative). The results are written to DATA_DIR as daily_headline_polarity.parquet

import create_firmday_score as cfs

cfs.main()

[CHECK] outputs parsed unique custom_id: 209,710
[CHECK] outputs skipped: empty_choices=0, empty_content=0, unparseable=4

[CHECK] mapping rows unique custom_id: 209,714
[CHECK] after merge outputsā†’mapping:

_merge
both          209710
left_only          0
right_only         0
[CHECK] rows after dropping missing ticker/entity/date/headline: 209,710 (dropped 0)

[CHECK] trading days in CRSP calendar: 1,087 (min=2021-10-01, max=2026-01-30)

[CHECK] rows after dropping missing trade_date: 203,954 (dropped 5,756)
[CHECK] final firm-day rows: 176,968
[CHECK] firm-day date range: 2021-10-01 ā†’ 2026-01-30
[CHECK] unique trading dates in firm-day: 1,087
Wrote: /home/tomhi/finmath/finm-32900/p05_lopez-lira_tang_2023/_data/daily_headline_polarity.parquet (rows=176,968)

scores = pd.read_parquet(paths["Daily Headline Scores"]) if paths["Daily Headline Scores"].exists() else pd.DataFrame()
print("Scores shape:", scores.shape)
display(scores.head(5).style)

if not scores.empty:
    summary = (
        scores["score"]
        .value_counts(dropna=False)
        .rename_axis("score")
        .reset_index(name="count")
    )
    summary["share"] = (summary["count"] / summary["count"].sum()).round(4)
    summary.sort_values("score", inplace=True, ignore_index=True)
    summary = summary.rename(index={0: "NO", 1: "UNKNOWN", 2: "YES"})
    display(summary.style)

Scores shape: (176968, 5)
  ticker date n_headlines score_sum score
0 A 2021-11-18 1 1 1
1 A 2021-11-23 2 1 1
2 A 2022-01-27 1 0 0
3 A 2022-02-16 1 1 1
4 A 2022-02-23 3 2 1
  score count share
NO -1 9296 0.052500
UNKNOWN 0 118923 0.672000
YES 1 48749 0.275500

Step 6. Portfolio Returns#

Pipeline step that generated this data

  • portfolio_daily_returns.parquet is generated by process:create_portfolios via create_portfolios.py.

In order to actually replicate the paper results, we need to translate the headline scores into trading signals and build the following daily-rebalanced portfolios as described in the paper:

  • Long-Short Portfolio: buying positive predictions and selling negative predictions when both legs have at least two firms; otherwise entering just one leg.

  • Long-Only Portfolio: buying only positive predictions.

  • Short-Only Portfolio: selling only negative predictions.

Moreover, to replicate Figure 5 from the paper, which shows the cumulative return of the Long-Short portfolio with additional restrictions as described in the paper (we are excluding transaction costs):

  • Not Small: market capitalization above the 20th percentile

  • Price > 5: close price greater than $5 on previous day

port = pd.read_parquet(paths["Portfolio Returns"]) if paths["Portfolio Returns"].exists() else pd.DataFrame()
print("Portfolio returns:", port.shape)
display(port.head(5).style)

Portfolio returns: (1087, 20)
  date n_neg n_neu n_pos n_total ret_long n_long n_short ret_short ret_ir_long ret_ir_short ret_ls_restricted ret_ls_not_small ret_ls_price_gt_5 ret_mkt_vw trade_long trade_short trade_ls ret_ls ret_ir_ls
0 2021-10-01 3 37 19 59 0.000000 0 0 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 True True True 0.000000 0.000000
1 2021-10-04 3 83 37 123 -0.001595 32 2 -0.001542 0.003042 0.051936 -0.004477 -0.004432 0.000915 -0.010693 True True True -0.003137 0.054978
2 2021-10-05 5 107 33 145 0.020661 29 5 0.026466 0.006686 0.001040 0.066331 0.066341 0.059321 0.006008 True True True 0.047127 0.007726
3 2021-10-06 3 113 45 161 0.009360 40 2 0.033815 0.008709 0.023499 -0.013591 -0.013591 -0.015007 0.011786 True True True 0.043175 0.032208
4 2021-10-07 3 93 43 139 0.001959 41 1 -0.054348 0.007832 0.114533 -0.063873 -0.062724 -0.059204 0.002413 True True True -0.052389 0.122366

Step 7: Generate Table 1#

Pipeline step that generated this data

  • table1_overnight_full_sample.csv is generated by process:create_table1 via create_table1.py

  • table1_overnight_paper_sample.csv is generated by process:create_table1 via create_table1.py

Once we have our portfolios constructed we can attempt to replicate Table 1 from the paper. Additionally, we will generate a version of the table that includes data through January 2026. The source of the portfolio data is portfolio_daily_returns.parquet as described above.

import create_table1 as ct1

ct1.main()

Wrote /home/tomhi/finmath/finm-32900/p05_lopez-lira_tang_2023/_output/table1_overnight_paper_sample.csv (rows=4)
Wrote /home/tomhi/finmath/finm-32900/p05_lopez-lira_tang_2023/_output/table1_overnight_full_sample.csv (rows=4)

Paper sample date range:
2021-10-01 00:00:00 to 2024-05-31 00:00:00

Full sample date range:
2021-10-01 00:00:00 to 2026-01-30 00:00:00

full_sample_table = pd.read_csv(paths["Table1 (Oct 2021 - March 2026)"]) if paths["Table1 (Oct 2021 - March 2026)"].exists() else pd.DataFrame()
paper_sample_table = pd.read_csv(paths["Table1 (Oct 2021 - May 2024)"]) if paths["Table1 (Oct 2021 - May 2024)"].exists() else pd.DataFrame()
display(printable_table(full_sample_table.head(5), "Full Sample").style)
display(printable_table(paper_sample_table.head(5), "Paper Sample Replication").style)

Full Sample Data: (4, 8)
Summary:
  > Trading Days:  1087
  > Firm-Day Observations:  176968
  Portfolio Initial Reaction Hit Rate (%) Initial Reaction Mean Return (%) Drift Hit Rate (%) Drift Mean Return (%) Drift Sharpe Ratio
0 Long-Short Portfolio 93.100000 4.381000 51.702000 0.165000 0.912000
1 Long-Only Portfolio 85.465000 3.004000 47.470000 -0.036000 -0.321000
2 Short-Only Portfolio 84.885000 1.392000 55.945000 0.187000 0.993000 
Paper Sample Replication Data: (4, 8)
Summary:
  > Trading Days:  670
  > Firm-Day Observations:  112092
  Portfolio Initial Reaction Hit Rate (%) Initial Reaction Mean Return (%) Drift Hit Rate (%) Drift Mean Return (%) Drift Sharpe Ratio
0 Long-Short Portfolio 92.836000 3.300000 52.239000 0.240000 1.458000
1 Long-Only Portfolio 84.179000 1.973000 48.507000 0.020000 0.195000
2 Short-Only Portfolio 85.629000 1.353000 54.491000 0.215000 1.214000

Replication Comparison#

actual_paper_table = pd.read_csv(paths["Paper Table Data"]) if paths["Paper Table Data"].exists() else pd.DataFrame()
display(printable_table(actual_paper_table, "Actual Paper Table").style)

Actual Paper Table Data: (4, 8)
Summary:
  > Trading Days:  670.0
  > Firm-Day Observations:  105742
  Portfolio Initial Reaction Hit Rate (%) Initial Reaction Mean Return (%) Drift Hit Rate (%) Drift Mean Return (%) Drift Sharpe Ratio
0 Long-Short Portfolio 93.280000 3.060000 58.060000 0.340000 2.970000
1 Long-Only Portfolio 83.280000 1.270000 50.900000 0.080000 0.780000
2 Short-Only Portfolio 79.400000 1.790000 53.580000 0.260000 2.010000 
actual = actual_paper_table

compare_cols = [col for col in paper_sample_table.columns if col in actual.columns]

actual_cmp = actual[compare_cols].set_index("Portfolio")
paper_cmp = paper_sample_table[compare_cols].set_index("Portfolio")

numeric_cols = [col for col in compare_cols if col != "Portfolio"]

diff_table = paper_cmp[numeric_cols].apply(pd.to_numeric, errors="coerce") - actual_cmp[
    numeric_cols
].apply(pd.to_numeric, errors="coerce")

display(diff_table.reset_index().style.format({col: "{:.4f}" for col in numeric_cols}))

pct_error_table = pd.DataFrame(index=actual_cmp.index)

for col in numeric_cols:
    actual_vals = pd.to_numeric(actual_cmp[col], errors="coerce")
    paper_vals = pd.to_numeric(paper_cmp[col], errors="coerce")
    pct_error = ((paper_vals - actual_vals) / actual_vals.replace(0, pd.NA)) * 100

    pct_error_table[f"{col} | actual"] = actual_vals
    pct_error_table[f"{col} | paper_sample"] = paper_vals
    pct_error_table[f"{col} | error (%)"] = pct_error

pct_formats = {}
for col in numeric_cols:
    pct_formats[f"{col} | actual"] = "{:.4f}"
    pct_formats[f"{col} | paper_sample"] = "{:.4f}"
    pct_formats[f"{col} | error (%)"] = "{:.2f}%"

error_cols = [c for c in pct_error_table.columns if c.endswith("| error (%)")]
display(
    pct_error_table[error_cols]
    .reset_index()
    .style.format({col: "{:.2f}%" for col in error_cols})
)
  Portfolio Initial Reaction Hit Rate (%) Initial Reaction Mean Return (%) Drift Hit Rate (%) Drift Mean Return (%) Drift Sharpe Ratio Trading Days Firm-Day Observations
0 Long-Short Portfolio -0.4440 0.2400 -5.8210 -0.1000 -1.5120 nan nan
1 Long-Only Portfolio 0.8990 0.7030 -2.3930 -0.0600 -0.5850 nan nan
2 Short-Only Portfolio 6.2290 -0.4370 0.9110 -0.0450 -0.7960 nan nan
3 Sample Summary nan nan nan nan nan 0.0000 6350.0000
  Portfolio Initial Reaction Hit Rate (%) | error (%) Initial Reaction Mean Return (%) | error (%) Drift Hit Rate (%) | error (%) Drift Mean Return (%) | error (%) Drift Sharpe Ratio | error (%) Trading Days | error (%) Firm-Day Observations | error (%)
0 Long-Short Portfolio -0.48% 7.84% -10.03% -29.41% -50.91% nan% nan%
1 Long-Only Portfolio 1.08% 55.35% -4.70% -75.00% -75.00% nan% nan%
2 Short-Only Portfolio 7.85% -24.41% 1.70% -17.31% -39.60% nan% nan%
3 Sample Summary nan% nan% nan% nan% nan% 0.00% 6.01%

Comparison Discussion#

As we can see from the above, some of our replication was fairly consistent with the papers results, specifically Initial Reaction Hit Rate of the Long-Short and Long-Only portfolios as well as the Drift Hit Rate of the Short-Only portfolio. Unfortuantely, everything else was substantially different.

This can almost certainly be attributed to one key fact: the GPT model specified in the paper, gpt-4-0314, was deprecated by OpenAI and no logner available. Therefore we opted to use the next most recent model gpt-3.5-turbo. Given the paperā€™s results suggesting the older the model the worse it would perform, it is no surprise that the mdoel we used largely performed worse.

Specifically, we note that our responses from OpenAI using the older model produced significantly more UNKNOWN responses than the papers model, i.e. ~67% vs 34.7%

import create_openai_responses_table as cort

cort.main()

  label  proportion_2021_2024_sample_period  proportion_full_sample  count_2021_2024_sample_period  count_full_sample
    YES                            0.268041                0.272443                          34874              57134
     NO                            0.060035                0.062133                           7811              13030
UNKNOWN                            0.671924                0.665424                          87422             139546

Wrote /home/tomhi/finmath/finm-32900/p05_lopez-lira_tang_2023/_output/openai_output_label_proportions.csv
Contents