nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Adding labels or categories for news

Mohamed_Magdy2026

I usually use this python code in the attached photo to extract news data for Egypt so that I can make my own analysis on financial news. However, this gives me tons of irrelevant news (e.g. FIFA World Cup news).

I need to modify it by including columns giving me RICs, categories or labels associated with these news (e.g. Macro, Political…etc), so that I can filter out redundant news.

1.PNG

Find more posts tagged with

python

news

#technology

Accepted answers

Jirapongse

@Mohamed_Magdy2026

Please refer to this LSEG Data Library for Python: News Pagination article.

The code can be modified like this:

def create_topic_dataframe(raw_data):
    data_array = []
    if 'data' in raw_data:
        news_data = raw_data['data']
    else:
        news_data = raw_data[0]['data']
        
    for story in news_data:
        topics = []
        rics = []
        for code in story['newsItem']['contentMeta']['subject']:
            if code['_qcode'].startswith('R:'):
                rics.append(code['_qcode'].split(':')[1])
            else:
                topic = concept_dict.get(code['_qcode'])
                if topic:
                    topics.append(topic)     
            
        data_array.append(
            {
                'StoryId': story['storyId'],
                'Headline': story['newsItem']['itemMeta']['title'][0]['$'],
                'FirstCreated': story['newsItem']['itemMeta']['firstCreated']['$'],
                'VersionCreated':story['newsItem']['itemMeta']['versionCreated']['$'],
                'RIC': ' '.join(rics),
                'SourceCode': story['newsItem']['contentMeta']['creator'][0]['_qcode'],
                'Topics': topics
            })
    df = pd.DataFrame(data_array)
    return df

def get_all_headlines(news_definition, df_list):
    _cursor = ""
    _response = news_definition.get_data()
    if _response.data.df.shape[0] > 0:
        df_list.append(create_topic_dataframe(_response.data.raw))    
    if type(_response.data.raw) == list:
        if "next" in _response.data.raw[0]["meta"]:
            _cursor = _response.data.raw[0]["meta"]["next"]
        else:
            _cursor = ""
    else:
        if "next" in _response.data.raw["meta"]:
            _cursor = _response.data.raw["meta"]["next"]
        else:
            _cursor = ""
   
    if _cursor != "":
        _news_def = news.headlines.Definition(
            query = "",
            count = 100,
            extended_params={
                "cursor":_cursor
            })
        get_all_headlines(_news_def, df_list)

Then, you can call the get_all_headlines function to retrieve all news headlines.

headlines_def = news.headlines.Definition(
    query = "Egypt",
    date_from = datetime.datetime.today(),
    date_to = datetime.datetime.today() - datetime.timedelta(days=2),
    count=100)
headlines_df_list = []
get_all_headlines(headlines_def, headlines_df_list)
pd.concat(headlines_df_list)

All comments

ujjawal_khandelwal

Hi @Mohamed_Magdy2026,
Thank you for reaching out!

The behavior you’re seeing is expected. Your current query is very broad, as it includes all news related to Egypt. Because of this, the results may feel too generic or overwhelming.

✅ How to improve your results:

To get more relevant insights, try narrowing down your query based on a specific theme or topic you’re interested in.

A good way to do this is by using the TOPICS app in LSEG Workspace, which helps you identify predefined topic codes.

Let's say you're interested in innovation-related news in Egypt, you can refine your query like this:

news_egypt = ld.news.get_headlines(

    query="Egypt AND Topic:INNVTN AND Language:EN",

    start="2026-01-01",

    end="2026-05-31",

    count=50

)


display(news_egypt)

Best practices:

Be specific → Add topics, sectors, or keywords
Use topic codes (e.g., INNVTN for innovation)
Keep count lower → Helps avoid hitting your data usage limits

You can also refer to the below article for detailed information about usage limits:

The Data Library for Python - Maximum Usage Reference Guide | Devportal

Please let me know if you found this helpful.
Thank you : )

Mohamed_Magdy2026

Hi @ujjawal_khandelwal

Many thanks for your helpful reply. I've applied your idea by identifying the topics I need (MCE,CMPNY), but still, I need to a add column labeling the Topic name for each news. In addition, I need another column telling me the associated RIC for that news if any.

I'd be grateful if you can give this code so that I can smoothly proceed into my analysis.

2.PNG

ujjawal_khandelwal

Hi @Mohamed_Magdy2026,

Most welcome :)

I don't think that is possible. The reason being - news is not always tied to a RIC. Plus a single news headline or story can have multiple tags.

The easy option is to add the RIC into your query. You can use AND or OR and based on it, the api will return the results.

I understand that if you want to fetch news for multiple RICs, how do I filter it later on. I would suggest doing it prior to making a call.

You can use a for loop and 'time' library to make a call for one RIC > pause > loop runs > fetch data for another RIC. I can share the code - will attach it soon.

Please let me know if this helps. Thanks:)

Jirapongse

@Mohamed_Magdy2026

You can use the News interface in the content layer to retrieve the raw news headlines.

For each headline, the raw content will contain the topic codes in the subject field.

 {'storyId': 'urn:newsml:newsroom:20260624:nNRA0zqr4z:0',

   'newsItem': {'_version': 0,

    'contentMeta': {'audience': [],

     'creator': [{'_qcode': 'NS:AGEECP', '_role': 'sRole:source'}],

     'infoSource': [{'_qcode': 'NS:AGEECP', '_role': 'sRole:source'},

      {'_qcode': 'NS:AGEECP', '_role': 'sRole:origProv'}],

     'language': [{'_tag': 'fr'}],

     'subject': [{'_qcode': 'G:V'},

      {'_qcode': 'G:D3'},

      {'_qcode': 'B:127'},

      {'_qcode': 'M:1WN'},

      {'_qcode': 'G:3G'},

      {'_qcode': 'B:261'},

      {'_qcode': 'R:AEIBy.MZ'},

      {'_qcode': 'B:131'},

      {'_qcode': 'B:262'},

      {'_qcode': 'B:1629'},

      {'_qcode': 'G:D'},

      {'_qcode': 'L:fr'},

      {'_qcode': 'G:1'},

      {'_qcode': 'G:Q'},

      {'_qcode': 'Meta:NewsRoom'},

      {'_qcode': 'P:5000419992'}],

To map these topic codes to the names, you can use the Machine Readable News - News codes file. You can use the following Python code to create a Python dictionary to map the news topic codes.

 import pandas as pd

# Load the Excel file
file_path = "409397.xlsx"

# Read all sheets
sheets = pd.read_excel(file_path, sheet_name=None, engine="openpyxl")

# Initialize dictionary
concept_dict = {}

# Iterate the News Topics sheet
for sheet_name, df in sheets.items():
    # Ensure required columns exist
    if sheet_name == 'News Topics':
        if 'conceptId' in df.columns and 'name' in df.columns:
            # Drop rows with missing values (optional but recommended)
            tmp = df[['conceptId', 'name']].dropna()
    
            # Update dictionary
            concept_dict.update(dict(zip(tmp['conceptId'], tmp['name'])))

# Example: lookup
print(concept_dict.get("G:D"))

Then, pull the news headlines and create a data frame.

import lseg.data as ld
from lseg.data.content import news

#Open a session
ld.open_session()

#Retrieve News
response = news.headlines.Definition("Egypt", count=100).get_data()


#Iterate all rows to create a data frame
data_array = []
if 'data' in response.data.raw:
    news_data = response.data.raw['data']
else:
    news_data = response.data.raw[0]['data']
for story in news_data:
    topics = []
    rics = []
    for code in story['newsItem']['contentMeta']['subject']:
        if code['_qcode'].startswith('R:'):
            rics.append(code['_qcode'].split(':')[1])
        else:
            topic = concept_dict.get(code['_qcode'])
            if topic:
                topics.append(topic)     
        
    data_array.append(
        {
            'StoryId': story['storyId'],
            'Headline': story['newsItem']['itemMeta']['title'][0]['$'],
            'FirstCreated': story['newsItem']['itemMeta']['firstCreated']['$'],
            'VersionCreated':story['newsItem']['itemMeta']['versionCreated']['$'],
            'RIC': ' '.join(rics),
            'SourceCode': story['newsItem']['contentMeta']['creator'][0]['_qcode'],
            'Topics': topics
        })
df = pd.DataFrame(data_array)
df

The output looks like this:

Mohamed_Magdy2026

@Jirapongse

Thank you for your reply

I have additional questions to ensure I fully understand the whole process:

1- The 1st lines of codes (1-23) don't yiled an output. It seems to be part of another code but I don't know it or how to use it

2- What is "409397.xlsx" file? Is it an output that would have been resulted from the 1st lines of codes (1-23)?

3- Can I specify start & end dates for these news?

Jirapongse

The first code block is used to create a Python dictionary for the news code used by the second part of code.
This is the news codes file. Sorry. I forgot to attach the link.
The news.headlines.Definition interface supports the date_from and date_to parameters.

Help on class Definition in module lseg.data.content.news.headlines._definition:


class Definition(lseg.data.content.news._news_data_provider_layer.NewsDataProviderLayer)

 |  Definition(query: str, count: int = 10, date_from: 'OptDateTime' = None, date_to: 'OptDateTime' = None, sort_order: Union[str, lseg.data.content.news.headlines._sort_order.SortOrder] = <SortOrder.new_to_old: 'newToOld'>, extended_params: 'ExtendedParams' = None)

 |

 |  This class describes parameters to retrieve data for news headlines.

 |

 |  Parameters

 |  ----------

 |  query: str

 |      The user search query.

 |

 |  count: int, optional

 |      Count to limit number of headlines. Min value is 0. Default: 10

 |

 |  date_from: str or timedelta, optional

 |      Beginning of date range.

 |      String format is: '%Y-%m-%dT%H:%M:%S'. e.g. '2016-01-20T15:04:05'.

 |

 |  date_to: str or timedelta, optional

 |      End of date range.

 |      String format is: '%Y-%m-%dT%H:%M:%S'. e.g. '2016-01-20T15:04:05'.

 |

 |  sort_order: str or SortOrder

 |      Sort order for the response. Default: SortOrder.new_to_old

 |

 |  extended_params: dict, optional

 |      Additional parameters to provide to the API.

 |

 |  Examples

 |  --------

 |  >>> from datetime import timedelta

 |  >>> from lseg.data.content import news

 |  >>> definition = news.headlines.Definition(

 |  ...     "Refinitiv",

 |  ...     date_from="20.03.2021",

 |  ...     date_to=timedelta(days=-4),

 |  ...     count=3

 |  ... )

 |

Please also refer to this example.

Mohamed_Magdy2026

@Jirapongse

Many thanks for illustrative response

One more question if you please: I noticed that that the number of news is capped at 100. Even when selecting small time interval (e.g. one day), it shows a maximum of 100. Is there a way that enables me to show these columns (RIC & Topics) for all the news?

Jirapongse

@Mohamed_Magdy2026

Please refer to this LSEG Data Library for Python: News Pagination article.

The code can be modified like this:

def create_topic_dataframe(raw_data):
    data_array = []
    if 'data' in raw_data:
        news_data = raw_data['data']
    else:
        news_data = raw_data[0]['data']
        
    for story in news_data:
        topics = []
        rics = []
        for code in story['newsItem']['contentMeta']['subject']:
            if code['_qcode'].startswith('R:'):
                rics.append(code['_qcode'].split(':')[1])
            else:
                topic = concept_dict.get(code['_qcode'])
                if topic:
                    topics.append(topic)     
            
        data_array.append(
            {
                'StoryId': story['storyId'],
                'Headline': story['newsItem']['itemMeta']['title'][0]['$'],
                'FirstCreated': story['newsItem']['itemMeta']['firstCreated']['$'],
                'VersionCreated':story['newsItem']['itemMeta']['versionCreated']['$'],
                'RIC': ' '.join(rics),
                'SourceCode': story['newsItem']['contentMeta']['creator'][0]['_qcode'],
                'Topics': topics
            })
    df = pd.DataFrame(data_array)
    return df

def get_all_headlines(news_definition, df_list):
    _cursor = ""
    _response = news_definition.get_data()
    if _response.data.df.shape[0] > 0:
        df_list.append(create_topic_dataframe(_response.data.raw))    
    if type(_response.data.raw) == list:
        if "next" in _response.data.raw[0]["meta"]:
            _cursor = _response.data.raw[0]["meta"]["next"]
        else:
            _cursor = ""
    else:
        if "next" in _response.data.raw["meta"]:
            _cursor = _response.data.raw["meta"]["next"]
        else:
            _cursor = ""
   
    if _cursor != "":
        _news_def = news.headlines.Definition(
            query = "",
            count = 100,
            extended_params={
                "cursor":_cursor
            })
        get_all_headlines(_news_def, df_list)

Then, you can call the get_all_headlines function to retrieve all news headlines.

headlines_def = news.headlines.Definition(
    query = "Egypt",
    date_from = datetime.datetime.today(),
    date_to = datetime.datetime.today() - datetime.timedelta(days=2),
    count=100)
headlines_df_list = []
get_all_headlines(headlines_def, headlines_df_list)
pd.concat(headlines_df_list)

Mohamed_Magdy2026

@Jirapongse

Thank you very much. Now I was able to get the job done

EXPLORE OUR SITES