Archive

Posts Tagged ‘Capital IQ’

Extending your Excel worksheet formulas via Python, for Capital IQ

17 March 2017 3 comments

Downloading bulk data from financial databases can be a tiring process. You may need panel data structured in a very specific way for analysis in a tool like Stata, and the data provider provides data in a different and limited way; perhaps for one company or one year at a time, leaving you to build it into the shape you need. Each product, like Bloomberg, Datastream, ThomsonONE.com and Capital IQ, provide a different Excel interface. Sometimes you can build a formula for each data item for each year for each company, then fill or copy it across or down to save time. This extended blog post will show you how to do this for the Capital IQ Office Plug-in, and how to save time extending it for panel data using a short Python script.

What kind of data are we looking at?

First, let’s consider the types of variables in turn. In no particular order:

  1. Entities: That is, Companies, Funds, Equities, Indices, Persons…
  2. Time: Usually one of Years, Quarters, Months, Days…
  3. Data items/types: Time series or line items such as Total assets, Turnover, Closing price…

If we consider any one variable or dimension at a time, this is easy; it’s just a list of identifiers or numbers.

  1. AAPL, MSFT, GOOG
  2. 1995, 1996, 1997
  3. TOTAL_ASSETS, NUM_EMPLOYEES, CREDIT_RATING

We can easily create a table to list any two of these dimensions simply.

For example: (fictional) figures for one company:

Item\Year 2001 2002 2003
Total assets ($m) 6.3 7.2 8.9
Net profit ($m) 2.3 4.7 6.1

Or, (fictional) figures for Year 2004:

Company\Item Employees (thousands) Credit rating
AAPL 50 AAA
MSFT 65 AA
GOOG 14 AAA

But if we want to look at three at once, it gets more complicated. More on that later.

Getting data from Capital IQ Office Plug-in

This post assumes you are familiar with using Capital IQ and its Excel add-in. If not, search for other posts on Capital IQ or read the hand-out guides we have written before proceeding.

In short, you can use an Excel formula to pull data from the Capital IQ servers to your spreadsheet. These can be created in the Formula Bulider menu, or typed out manually. For example, to get total assets, the formula will be =CIQ(“IQ18671”, “IQ_TOTAL_ASSETS”, “FY2012”) if the company identifier is IQ18671 and the selected date is financial year 2012. [Note that the double-quote characters appear ‘curved’ here in WordPress and you’ll need to retype them clean in Excel.]

Tip: The data retrieved will only load on a PC with the Capital IQ Excel add-in set up; you will need to copy then paste-as-values, or save as CSV to keep the data for use elsewhere.

Using the fill down/across command

The formula includes several arguments — the options separated by commas — which are “hard coded” – the values for which company, date or item are included as text strings.

Basic Capital IQ formula

The CIQ formula in the yellow cell is shown shown in the formula editor. It references the company in A2, the item “IQ_TOTAL_ASSETS” and the date “FY2012”.

You can replace these arguments with cell references, to a cell that contains the same text, or part of it.

Replacing the item with a cell reference

The yellow cell’s formula now includes a cell reference B$1 for the item type. Note the dollar sign locks the row number when we fill the formula down.

This will allow you to use the Excel feature fill down or fill across, to extend the formula down or right.

Fill the formula down and right

The cell reference for date follows the text “FY” then an ampersand, as the cell B2 contains just the year number. Note the dollar sign in B$2, and now also in the company cell reference $A3, so that the formula fills right correctly.

It will work for any argument, including item or date, although you need to take more care with date. In the example above we use year, and Capital IQ expects the year number to follow the letters “FY” for Financial Year (or “CY” for Calendar Year). We must therefore use the ampersand ‘&’ character to join the strings.

Adding the third dimension

So far we have looked tables to represent two of our three dimensions. There are six permutations for this:

  • Companies down and dates across, for one item
  • Companies down and items across, for one date
  • Dates down and items across, for one company
  • and each of those transposed the other way around.

So if we want to show all three, we have to essentially stack more tables across or down. There are many more ways to arrange the data like this, such as:

  • Companies down, items across, dates across
  • Companies down, date across, items across

It might be easier to show these in a dummy table:

Company ID Item1-2001 Item2-2001 Item1-2002 Item2-2002 Item1-2003 Item2-2003
comp1  a  b  c  d  e  f
comp2  g  h  i  j  k  l

Or

Company ID Item1-2001 Item1-2002 Item1-2003 Item2-2001 Item2-2002 Item2-2003
comp1  a  c  e  b  d  f
comp2  g  i  k  h  j l

You might prefer to stack into panels as below (this is like reshaping from wide to long in Stata). We have to repeat the first column (company ID) for each date… do not be tempted to use merged cells here! If you did, you won’t be able to use data filters, Excel tables or correctly export to other programs.

Company ID Year Item1 Item2
comp1 2001  a  b
comp1 2002  c  d
comp1 2003  e  f
comp2 2001  g  h
comp2 2002  i  j
comp2 2003  k  l

Often we start with a list of thousands of companies. It would be very time consuming to manually insert extra lines for each year after each company! This is where the script comes in…

Extending the process with a Python script

In order to perform a predictable and repetitive task, it can be a good idea to write a short script instead. The method below uses a combination of Excel and an external programming language, Python, and some human intervention. (I assume you have Python installed – the University PCs tend to have it, sometimes as part of SPSS.)

  1. Start with a column of the company IDs in Excel, and save this as a text file. You can do this by copying the column and pasting it into Notepad, remembering to delete any header rows. (You can instead paste it to a new workbook in Excel and save as Text (Tab delimited) (*.txt).) The file must be named ids.txt for the script to work.

    List of company IDs

    Save the list of IDs to a text file ids.txt

  2. Now for the Python script, which is shown at the end of the post for you to copy. Copy and paste this into a text editor like Notepad, although Notepad++ is included on University of Manchester PCs and is much better. If the list of years {1995, 1996, 1997} is not right for you, change it here! Save this as script01.py or similar – note the file extension is .py and not .txt – this is a Python script. Save it to the same directory as the ID file.

    Python code, showing whitespace

    Notepad++ showing Python code (you can copy-paste it from the end of this blog post), showing whitespace/tabs in orange. Python scripts must have consistent line indents, tabs or spaces.

  3. Run the script. To do this, open Windows Explorer, navigate to the folder you have your two new files in and hold Shift + Right-click then choose “Open command window here”. A black command window will open, type “python script01.py” (no quotes) and press enter. Hopefully no error message will show, you’ll get no message if it was successful.

    Run the script

    Shift+Right-click to get “Open command window here”, type “python script01.py” and press Enter. File template.txt is created.

  4. Find the newly created file, template.txt, and open it in Excel. There are several ways to open tab-delimited text files in Excel, any should do.
    Open .txt file in Excel, part 1

    Excel, open file, select type “Text Files (*.prn; *.txt; *.csv)” and choose template.txt. Accept any warning messages.

    Open .txt file in Excel - part 2

    This is a delimited text file, with Tab delimiters only.

  5. You can type the item names in C1, D1, E1 etc., such as “IQ_FINISHED_INV” and “IQ_TOTAL_ASSETS” (no quotes). You can type the template Capital IQ formula in C2, remembering to replace the literal arguments with cell references, and remembering to lock some of the references with dollar signs. For example, =CIQ($A2, C$1, “FY”&$B2)

    Fill the template columns and first formula

    Complete the green cell column headers (item codes). Fill in the yellow cell formula with correct cell references and locks, here it is =CIQ($A2, C$1, “FY”&$B2)

  6. Save the workbook as regular Excel file (*.xlsx). Now select the cell, hover the mouse over the bottom-right corner for the little black square icon, and drag the icon to the right to select that row in all the columns you need. The formula will copy across, and hopefully correctly. You may need to edit it and try again if the dollar sign locks went wrong. Now fill the formula down (just a few rows at first to test, then all the way down by double-clicking the black square icon).

    Final fill out

    Save before you start! Fill the formula right, then down a few rows, then further. Don’t just fill all the way to the bottom if there are thousands of rows, this might overload Capital IQ.

The Python code, which should have indents on the lines in each for loop:

fin = open('ids.txt')
fout = open('template.txt', 'w')
years = {"1995", "1996", "1997"}
# header
fout.write("ID\tYear\n")
#body
for line in fin:
   id = line.rstrip()
   for year in years:
       lineout = id + "\t" + year + "\n"
fout.write(lineout)
fin.close()
fout.close()

Moving on

  • The script is not the best example of Python, but it works! For example, the list of years should really be done better.
  • You could extend the script to include the items and Excel formulas, but for a one-off script, it is probably quicker to do some of the work in Excel by hand (using the fill function).
  • You could chose any other programming language over Python, including VBA which is built into Excel.
  • You could apply this technique to other databases with an Excel interface such as Bloomberg.

Related posts

Company Employee Data Using Different Databases

24 February 2016 1 comment

How many employees work for a company?

This would appear to be a simple question. But perhaps not…

Although the company in question will have a more exacting answer as to values at any particular point in time, this would be considered commercially sensitive information. Therefore, values are typically made available at a single point in time, such as the end of the company’s reporting period. This is then accessible via a number of business databases.

Unfortunately, if you look in different databases, you tend to get different answers – for example, looking at Tesco PLC, which is a publicly quoted UK company, a number of values are specified depending on the database consulted:

 

Bloomberg Professional

This shows 386,086 employees and helpfully specifies a date point of 28 February 2015, but displayed in US date format [02/28/15]. This reflects Tesco’s reporting period dates – essentially the start of March to the end of February.

Bb Tesco EE

 

Thomson ONE.com

This gives a different value [506,984] but specifies the same date – the end of the reporting period.

T1 Tesco EE PDF

 

Datastream (Thomson Reuters)

The datatype is ‘WC07011 – Employees’ and the definition notes: Employees represents the number of both full time and part time employees of the company. It excludes: Seasonal employees, Emergency employees. Footnotes:  D. Average employees.  This gives a value of 506,984.

Datastream Tesco EE

 

Fame

This specifies a value of 386,086.

Fame Tesco EE

 

Capital IQ

When searching for Tesco PLC, the ‘Number of Employees’ is stated as: 517,802.

Capital IQ Tesco EE

 

Company Annual Report

Perhaps one way to obtain clarity is to look in the ‘Tesco Annual Report and Financial Statements 2015’, accessible through Thomson Research or PI Navigator databases, for example.

Page 20, shows the following breakdown, for a total of 517,802 (220,257 + 297,545).

ARS 2015 Tesco EE p20

If you were to stop at this point in the Annual Report, you would miss further details. Hence, on page 100, there are additional details:

 

ARS 2015 Tesco EE p100

This helps to show which values the various databases have selected. Thus, Capital IQ has used the combined values [220,257 + 297,545 = 517,802] from page 20 of the Annual Report, in contrast to the other databases.

Thomson ONE.com/Datastream have used the ‘Average Number of Employees’ measure [506,984] and Bloomberg/Fame have used the ‘Average Number of Full-Time Equivalents’ [386,086] for 2015 – that is, the end of the reporting period, specifically 28 February 2015.

 

The lesson to be drawn when using data from a particular database – always quote the source, to ensure clarity and remove uncertainty. In the case of Employee data, the company’s Annual Report is the ultimate source.

However, the difference in total values between page 20 [517,802 (220,257 + 297,545)] and those quoted on page 100 [506,984 and 386,086] is a mystery. Specifying a page number in your reference is therefore recommended!

 

Bloomberg Professional, Datastream, Thomson ONE.com, Thomson Research, Fame and Capital IQ are databases available to current students and staff of The University of Manchester.

 

From ISIN codes to Compustat (via Capital IQ using GVKEY, CUSIP and Name)

17 December 2015 Leave a comment

Given a list of North American company ISIN codes from the Markit credit default swap database (via a lookup database), you may want to join in quarterly company fundamentals from Compustat in WRDS. Compustat does not accept ISIN codes; you have to use an intermediate source such as Capital IQ and one or more intermediate data types (GVKEY, CUSIP and Company Name).

Note, you must be using a PC with the Capital IQ Excel add-in set up. This is in the Library Finance Zone.

Method

  1. Use Capital IQ Office Plug-in (CIQ) with a column of ISIN codes in Excel. Put these in column A.
  2. In the next column, B, get the values of column A with a prefix of “I_” so CIQ can interpret them as ISINs. Cell B2 contains =”I_”&A2
  3. Next get the GVKEY code in column C with a CIQ formula. Cell C2 contains =CIQ(B2, “IQ_GVKEY”) — the output will need cleaning up to remove the “GV_” prefix and any multiple matches.
  4. Then the CUSIP codes in column D. Cell D2 contains =CIQ(B2, “IQ_CUSIP”) — this may not work well, I used the Compustat helpdesk here to get these.
  5. Finally from CIQ, the company names. Column E. Cell E2 contains =CIQ(B2, “IQ_COMPANY_NAME”)
  6. Create a new plain text document with the values of column C (GVKEY) and upload this to Compustat in the usual way.
  7. For the gaps, repeat with the values of column D (CUSIP).
  8. For any final gaps, use the Manually enter company codes > Code Lookup tool in Compustat, typing in the first part of the names to obtain any extra GVKEY codes one by one and repeat step 6.
  9. Stitch together the various Compustat output files and check that all the companies are present without duplicates.

Consider that CUSIP codes can change with time, so use them with caution. Remember to keep any column of CUSIP codes with Text format, never General, Number or Scientific.

Using business databases with Microsoft Office 2016 for Windows

13 October 2015 Leave a comment

Last month saw the release of the latest version of Microsoft’s ubiquitous Office suite for Windows, named version 2016. It brings new features and a design more in line with that of versions of Office released for iOS and Android. Many students beginning their studies this month will arrive at university with this version of Excel on their computers.

This is not a concern for users of the most specialist financial databases at The University of Manchester Library as the databases are installed on cluster PCs which will remain on Windows 7 with Office 2010; it may become a matter of concern for those at other institutions who are responsible for PC updates in the coming months.

I have run a few tests installing Thomson Reuters Datastream Advance for Office, Capital IQ Office Plug-in and Thomson Reuters Spreadsheet Link on Windows 7 and Windows 10 systems with Office 2016, specifically to check if their Excel add-ins work correctly. (I was not able to test Bloomberg due to its more limited restrictions.)

Datastream in Excel 2016 for Windows 7

Datastream in Excel 2016 for Windows 7

Datastream

Datastream in Excel 2010: The version of Datastream that is most used at The University of Manchester is Datastream Advance for Office (DAO), officially supported by Office 2007 and 2010 32-bit versions and no newer. (There is a version for 64-bit Office which I have not tested, called Datastream for Office (DFO), and a Datastream charting add-in, also not tested.)

Datastream for Excel 2013 in Windows 10Datastream in Excel 2013: Although Office 2013 is not officially supported for DAO, I managed to install it on a Windows 10 32-bit tablet and got a Request Table to work. (Tip: save the Request Table document the first time it asks you to, otherwise you will get into trouble with macro-enabled filetypes later.) I have seen the previous version of DAO, designed for Office 2003, working with Office 2013 32-bit on Windows 8.1 64-bit (sorry, screen shot not available).

Datastream install fail: AFOHelpDatastream in Excel 2016: Although Office 2016 is also not officially supported for DAO, I managed to install it on a Windows 7 32-bit PC and got a Request Table to work. I could not install it on a Windows 10 64-bit PC, using Office 2010 or Office 2016; it gave error messages such as “Compile error in hidden module: AFOHelp”.

In summary: Datastream works for me in Office 2016 with Windows 7 32-bit and possibly in some other situations but more testing is required.

Capital IQ

CIQ Excel 2016 data failI had a mixed but unsuccessful experience using the Capital IQ Office Plug-in. On a Windows 10 PC where I had already installed the software to use with Office 2010, running the Excel add-in on Excel 2016 would let me load the Capital IQ ribbon tab, access the template library, but it would not download any real data. I tried the usual troubleshooting methods but could not get the data to work.

CIQ Office 2016 install failWhen I tried to install Capital IQ Office Plug-in on Windows 7 where Office 2016 was installed, I got an error message that I could not work around: “Unsupported Office Version”. I guess Office 2016 is too far ahead of Office 2003 for it to recognise, even though I had Office 2010 installed as well.

I have not tried to contact their help desk yet; perhaps there will be a newer version released to address this. I am hopeful for this with Capital IQ; there will not be an update for either of these Thomson Reuters products tested today.

Thomson Reuters Spreadsheet Link

TRSL Office2010 Windows 10 failUnfortunately I was unable to run Thomson Reuters Spreadsheet Link (TRSL) in Office 2016. It is not officially supported in anything newer than Office 2010 so I was not expecting it to start working again in Office 2016, but running on the Windows 10 PC Office 2010 showed part of the Excel add-in loading; perhaps it had worked on that same PC a few months earlier.

TRSL in Office 2016 Windows 7 install failWhen I tried to install TRSL on a Windows 7 PC with Office 2016 installed, I got a message like with Capital IQ, stopping the installation from taking place at all. In short, it doesn’t work.

Summary

Many of the archive specialist database platforms that researchers use are dependent on old and ageing pieces of software. It is asking a lot to expect software written over five years ago to continue working in successive releases of the Windows operating system, and an even harder challenge to make it work within multiple Office environments on each of those platforms. Sometimes these old applications would require major redevelopment to work, sometimes just a tiny tweak. If that tiny tweak requires a disbanded team and development environment to be resurrected, it’s just as unlikely to be fixed as a major job. Sadly, the Thomson Reuters Excel add-ins fall into one of these categories. We will have to wait until Eikon is ready for wider use in education, hopefully before the Windows 7 operating system reaches the end of its life for receiving security updates in January 2020.

See also: Using business databases with Microsoft Edge on Windows 10

Insider trading

16 July 2015 Leave a comment

Definition of Insider Trading (from Investopedia):

The buying or selling of a security by someone who has access to material, nonpublic information about the security.

Companies in any country

Use ThomsonONE.com

  • Search for the company you want.
  • Click on the Company Analysis > Ownership folder, and select Insider Holdings or Insider Transactions report.
  • Click “Click here for full history” for historic holdings (older than 24 months).

Use Capital IQ

  • Search for the company you want.
  • On the long menu down the left, choose Investors > Public Ownership, then the Insider Trading tab.
  • Only the last 12 months are available.

Use Bloomberg (available in the Finance Zone and Precinct Library)

  • Try Bloomberg function PHDC for a specific company (e.g. VOD LN <F8> PHDC <GO>) and select Insiders, sorted by size.
  • You can also look up the biographies of executies/directors (BIO) and then choose Reported Holdings.
  • For stocks that have experienced the most purchases and sales by insiders, type: INSD and hit Go. Use the fields at the top of the screen to select country, time period, type of transaction. Click on a stock to display a graph of its historical prices with date of insider transactions.
  • For more info, type: insider and hit the green Help key <F1>.

Companies in the U.S. only

Use WRDS

  • Choose Thomson Reuters database, and then Insiders Data. Search by company ticker. Check the variables you want in your report.

Others

  • Insider filings with the U.S. SEC are available through Edgar. Enter your company name or ticker symbol or CIK and Include Ownership forms.
  • Insider transactions are available by issuer and reporting owner. Take a look at the website SecForm4.Com

Interested in finance? Test drive a database

23 September 2014 Leave a comment

Your interest in finance and financial markets could start with your university course, your career aspirations, intellectual curiosity, or a combination of all three. Whatever the reason, take one of the specialist financial databases for a quick test drive and get a glimpse of the world from the perspective of a finance professional.

Current students, and staff, at the University of Manchester have access to Bloomberg Professional from Bloomberg, Thomson ONE.com from Thomson Reuters, and Capital IQ from Standard & Poor’s.

Bloomberg Professional is a great database for getting the professional experience – you just have to think of a large, quoted company, find it and start to browse around. Bloomberg is only available on specific PCs so you have to go to the library.

Once you have Bloomberg running, type the company name in the command line, select the auto-matched series, and choose DES (description) for your command. The screenshot below shows the result for Samsung.

Bloomberg overview description of Samsung (click to enlarge)

Bloomberg overview description of Samsung (click to enlarge)

If you are unsure where to go on your test drive there are various Bloomberg posts that could give you inspiration.

Thomson ONE.com from Thomson Reuters is not quite as eye-catching as Bloomberg but it does have a web interface, so you can use it from any PC where you have Internet Explorer as your browser and can run the VPN software to access university resources.

Use the search box to find a large, quoted company by name and you get a standard overview report. Explore the different sections of the report and then the different tabs. The following screenshot is for Amazon.

T1com-Amazon

Thomson ONE company overview of Amazon

There are various Thomson ONE.com posts to give you tips. In addition, you can also use one of our library guides to help get started.

Capital IQ, like Thomson ONE.com, has a web interface – you need VPN and a special username and password. Getting started is the same – search for a large company by name and start with the default overview report. With Capital IQ the large list of options to explore appears as a list of headings on the left.

capitalIQ-walmart

Capital IQ company overview of Wal-Mart (click to expand)

This screenshot is for Wal-Mart Stores (who own the Asda in the UK).

When exploring any of these databases it is best to start with a large company as more information will be available, in general large, quoted US companies as the most studied by professionals are the best.

Ownership data from Capital IQ

From the research enquiries of students gathering data for their dissertations we know that data on company ownership can be difficult. Recent investigations suggest that Capital IQ from Standard and Poors (S&P), though not perfect, may be a good choice for many research projects.

The usual way to use Capital IQ is to use a screen to select the companies of interest, then add some additional variables, and finally download the results as an Excel file.

The screenshot below shows a company screen with two criteria:

  1. Companies with their primary listing on the London Stock Exchange (LSE)
  2. Companies with a market capitalisation (aka market value) greater than 1,000 million UK £s on 31 December 2013
Capital IQ company screening (click to expand)

Capital IQ company screening (click to expand)

This gives a total of 268 companies – since this is a test we want to be well within the Capital IQ download limit.

Next use the Customize Display Columns tab to add extra information to the results.

There is a good selection of variables that can be selected from the ownership section. For this test we choose:

  • CEO – Number of shares owned [Latest Quarter – 2]
  • CEO – Value owned [Latest Quarter – 2]

The only options for historical data are in terms of latest quarter – the oldest being latest quarter – 40. Selecting the CEO value owned column the results can be reordered to show which CEO’s of large LSE listed companies have the most wealth in their company’s shares.

Capital IQ ownership results (click to expand)

Capital IQ ownership results (click to expand)

Downloading the results is just a question of selecting Excel and go in the export section just below the screening results header, and waiting while Capital IQ formats the results and returns a download link.

The selection of Latest Quarter – 2 was an attempt to choose the Q4 2013. However, advice from the Capital IQ customer help desk suggest that this will give Q3 2013. The ownership data is updated when the company files the relevant data, usually 4-6 weeks after the end of a quarter. On 16 July we are just in Q3 but the Q2 data will not be available yet. This makes the Latest Quarter Q1 2014, and so Latest Quarter – 2 is Q3 2013.

 Alternatives:

There is company ownership information available in other databases, but they do not handle the combination of a list of companies and historical data as easily as Capital IQ

  • Thomson Reuters 13F data from WRDS (only data for  US companies)
  • Bloomberg – good for the ownership of one company
  • Thomson One.com –  detailed ownership for worldwide companies but with restrictions on the number of companies that can be investigated at one time, and results often need significant reformatting.
  • Fame, for UK and Irish companies, and Amadeus, for large Eur0pean companies, – can work with lists of companies but variables restricted to the owners holding the most shares.

Company ownership information may only be available for the last few years, or if historical information is available then it may be restricted to the largest shareholders.

Acknowledgements are due to Phil Reed and Xia Hong who explained how to use Capital IQ and noticed to availability of ownership data.

Categories: company information Tags: