I am trying to access historical bid/ask prices for a list of RICs (bonds). I am querying them in batches of 500 RICs at a time.
Some batches return an error code -1 with the message 'headers'. When this occurs, the entire batch is treated as invalid, and none of the observations from that batch are returned.
For example, if I run 20 batches, only 2 batches may have zero header issues. This results in 2,000 observations (500 RICs × 2 batches × 2 fields: bid and ask). All data from batches with header errors are lost, even though most observations in those batches are valid.
So in summary:
- Problem: Most of the data in the error-containing batches is actually fine, but the batch-level error causes it to be dropped entirely.
- Request: How can I modify my query or process so that I can keep the valid observations from batches that contain header errors, instead of losing the entire batch?
I'm pasting the code I'm using below. Thanks in advance.
#Get historical bid/ask quotes
#Setting count to 300 = 25 years to capture all data (this is when the series starts)
hist_bidask = []
#Running in batches to avoid timeouts/data issues.
batch_size = 500
for i in range(0, len(ric_list), batch_size):
batch = ric_list[i:i+batch_size]
try:
df_batch = rd.get_history(universe=batch,fields=['BID', 'ASK'], interval='monthly',count=300)
if df_batch is not None and not df_batch.empty:
df_batch = df_batch.T #transposing
hist_bidask.append(df_batch)
else:
print(f"Empty response for batch {i//batch_size + 1}")
except Exception as e:
print(f"Error in batch {i//batch_size + 1}: {e}")
print(f"Problematic RICs: {batch[:5]}...") # Show first 5 RICs
continue
#Concatenate into a dataframe (hist_bidask is a list)
hist_bidask = pd.concat(hist_bidask, ignore_index=False)
