How to Aggregate Universe Returns When Calculating Factor Returns

hiro_shiono

To calculate factor returns using TOPIX as the universe, I attempted to retrieve the stock prices of the index constituents. However, the volume of data is too large and I am hitting the daily data retrieval limit. I would appreciate any suggestions on a better approach.

I am currently downloading the data using the following process, but I encounter the limit at step 2:

1. Retrieve the RICs of TOPIX constituents as of a month-end date (for example, the end of August).
2. Retrieve the stock prices for those RICs for the following month (for example, September 1–30) and save them in a DataFrame.
3. Calculate returns directly from the stock prices stored in the DataFrame.

In addition, since there are approximately 1,600 constituents, I receive an error indicating that the request is too large when I attempt to retrieve all data at once. I therefore split the request into chunks of 50 RICs. However, some specific chunks return errors and no price data is retrieved.

I would be grateful for any advice on how to handle this more efficiently or reliably.

The actual code is as follows.

First, retrieve the universe:

df_tpx = ld.get_data(
    universe=['0#.TOPX'],
    fields=[f'TR.PriceClose(SDate={target_date},EDate={target_date})']
)

ric_col = df_tpx.columns[0]
rics = list(df_tpx[ric_col].dropna().unique())

Pattern 1:

df_history = ld.get_history(
    universe=rics,
    fields=['TRDPRC_1'],
    start='2026-07-31',
    end='2026-08-31',
    interval='1D'
)

Pattern 2:

start_date = '2026-07-30'
end_date = '2026-08-31'
chunk_size = 50
long_dfs = []

for i in range(0, len(rics), chunk_size):
    chunk_rics = rics[i:i + chunk_size]

    try:
        df_history = ld.get_history(
            universe=chunk_rics,
            fields=['TRDPRC_1'],
            start=start_date,
            end=end_date,
            interval='1D'
        )

        if df_history is not None and not df_history.empty:
            df_l = df_history.stack().reset_index()
            df_l.columns = ['date', 'ric', 'close']
            long_dfs.append(df_l)

    except Exception as e:
        print(
            f"Error occurred for chunk {i} to {i + len(chunk_rics)} "
            f"(skipped): {e}"
        )

Jirapongse

@hiro_shiono

Thank you for reaching out to us.

First of all, the code that retrieves the RICs of TOPIX constituents as of a month-end date looks incorrect. That code is used to retrieve historical price (TR.PriceClose) of the current TOPIX constituents.

To get, the historical constituents of TOPIX, the code should look like this:

df = ld.get_data(
    universe = ['.TOPX'],
    fields = ['TR.IndexConstituentRIC'],
    parameters = {'SDate':'2026-08-31'})
rics = df['Constituent RIC'].to_list()
len(rics)

However, please contact the content support team directly via LSEG Support to verify this TR.IndexConstituentRIC field.

Next, regarding the ld.get_history, using chunks is the appropriate solution when requesting a lot of items. However, you may need to add a retry logic to re-request data in case of there is a temporary issue in the backend service, as mentioned in this discussion.

Moreover, when retrieving historical real-time data (TRDPRC_1), it is important to understand how the LSEG Data Library processes requests internally. The historical pricing API endpoint used by ld.get_history() does not support batch requests for historical real-time data. Instead, each request can retrieve data for only a single RIC. The library manages this limitation transparently by issuing individual requests on behalf of the application.

As a result, when an application requests historical data for multiple instruments, the library breaks the request into chunks and submits separate requests for each RIC. If a chunk containing 50 instruments fails and the application retries that chunk, the library must resend all individual requests required to retrieve data for those 50 instruments. This behaviour can be inefficient and may significantly increase the overall processing time, particularly when working with large instrument universes.

A possible alternative is to request data one RIC at a time, allowing retries to target only the failed instrument rather than an entire chunk. Another option is to use the historical pricing asynchronous interface, which enables applications to submit one RIC per request while still processing multiple requests concurrently. This approach can provide finer-grained error handling and improve efficiency when retrying failed requests.