Three strategies

A system working with hyperspectral data has three resources to spend:

• Disk
• Time
• RAM

And there are three ways to handle derived products:

1. Derive on demand. Compute it when needed, use it, throw it away. Costs time.
2. Cache in RAM. Compute everything once at startup and hold it. Costs RAM.
3. Cache to disk. Compute once, store it, read it back. Costs disk.

Let’s go through them, worst first.

Cache in RAM

Many of my processes need both the smoothed and the normalised data. We could compute it for a whole hole at startup: a one-time slowdown, then fast iterative work and no wasted disk.

For an 80-box hole at ~300 MB per normalised dataset, that is 24 GB of RAM, before any algorithm allocates a single intermediate array, and before you meet a hole longer than average.

I do not see a >64 GB machine in my professional near future, never mind >128 GB. An 8 GB laptop is probably standard for the average geo-about-town. So this one is out.

Derive on demand

This is the option that deserves a proper look. It costs no disk, and its RAM footprint is small, because you only hold one box at a time:

Add the intermediates of a feature extraction and you are still under 1 GB. That is nothing, so RAM is not the problem here. Time is.

Here is the call signature for feature extraction in CoreSpecViewer:

def Combined_MWL(savgol, savgol_cr, mask, bands, feature, technique='QUAD',
                 use_width=False, cached_arrays=None):

It takes savgol (the smoothed reflectance) and savgol_cr (the normalised reflectance). Mask and bands have to be stored regardless. So, starting from cropped reflectance:

In this straightforward path, caching saves about 40%, which is real, but a slow feature extraction is still slow.

You will never run a feature extraction once. You pick parameters, look at the result, change a threshold, try a different fitting method, look again. You flick between boxes in the viewer, and you build downhole plots. Every one of those is an access.

Derive on demand pays the derivation cost on every access. Caching pays it once, and every access after that costs a disk read.

Ten iterations of tuning across one hole means 800 box accesses. At ~11 s of re-derivation each, that is nearly 2.5 hours spent recomputing data that has not changed. The cached version spends about 13 minutes reading it back. In exploratory, iterative work, the work this software exists for, the number of accesses explodes and the gap explodes with it.

So, disk

Disk caching vastly increases the access speed for derived data, but what pushes it over the edge into the absolute preferred option?
Memmapping.

Because everything lives on disk as arrays, CoreSpecViewer can memory-map it. That lets me have a 1 km hole “open” (over 150 GB of data) on an 8 GB RAM machine. Arrays are paged in as they are read and dropped back to the memmap when I am done with them.

You cannot memmap an array that does not exist yet. A whole-hole view built on derive-on-demand would mean deriving the whole hole, which drops us straight back into the RAM problem.

Caching everything looked wasteful when I started, since it nearly doubles the storage. But of the three resources, disk is by far the most abundant and the cheapest. A terabyte costs less than an afternoon of a geologist’s time.

What looked extremely wasteful is actually spending the abundant resource to protect the scarce ones.

“Conclusions”…

This conclusion depends on my stack.

“You fool!”, I hear you cry, “you wrote it in Python! Of course it is slow!”

You would not be wrong. Derivation could be pushed into compiled kernels, rewritten in another language, made blazingly fast. If continuum removal took one second instead of ten, the access-count argument would shrink a lot and I would revisit this. Maybe the commercial software world has already solved it. But Python gives me too much (NumPy, SciPy, hylite) for me to move away from it, and with this stack, disk wins.

In a nod to the fact disk caching doubles the storage CoreSpecViewer has an archive file type for long-term storage. This is the minimum dataset needed to deterministically reproduce a fully analysed box. Cache everything while you are working, and archive the minimum when you are done. The rehydrate process is the time penalty up-front when you open an archive, then you are back to working iteratively on memmaps.

The Original Linkedin post had some good points made by commenters that are worth reading, especially if you made it all the way through this!

Opening raw data

CoreSpecViewer exposes a number of data open options through it’s load dialogue.

Six different load modes

• Open processed dataset
• Open hole directory

These two refer to CoreSpecViewer’s own created data files, these files are what get saved after you have worked with a scan in the software.

• Open raw dataset from Lumo output directory
• Open raw dataset from individual ENVI files

These are for raw data from core scans. CoreSpecViewer will expect at least six files.

As I built CoreSpecViewer to work with data from the Specim Lumo acquisition software, clicking “from Lumo output directory” just requires the whole output directory, and all of the file discovery is automated and the metadata is parsed nicely.

If your data was acquired any other way, or your directory structure got mangled, the “from ENVI files” allows you to browse to each required file. The XML will be brute force parsed, but maybe not effectively, and CoreSpecViewer will prompt for the required metadata (hole id, box number, from depth, to depth).

• Open reflectance data

This button allows you to open a core scan that has already been converted to reflectance and stored in an ENVI file. It will also support existing masks (depending on the format).

• Open archive file

This is another file type produced by CoreSpecViewer, the need for which is discussed in this note

Pre-processing raw data

With hyperspectral data, you cannot just open the file and start interpreting the data, that would be far too easy. The cameras themselves will usually handle some geometric corrections, any spectral binning and some other correction before you get any data.
That just leaves correcting the radiance data to reflectance, which is why 6 files are needed.

BUT! CoreSpecViewer handles all of this for you! If you want to see the implementation it is here

Simply find your directory, or selection of files, click load and voilà

As this is python application, all of the details of the reflectance correction performed are in the source and parts of the configuration is adjustable in the UI. All of the configuration is adjustable in the source code.

This raw data can be cropped in the UI, a mask can be created using a variety of methods, and the pre-processing pipeline completed with a few button clicks:

This image is fully interactive, and individual pixel spectra can be examined using mouse clicks.

So far, so easy, but we haven’t actually learned anything about our core yet.

CoreSpecViewer has a number of tools to actually extract information from the core. I am not familiar with the geology of Alberta, or this drillhole, I am just using free data selected at random so…these interpretations will be less valid than those of a geologist who knows the geology!

Early exploration

I will perform a quick unsupervised clustering to see what variation my core has:

I used the default settings of 5 clusters for 50 iterations (these will obviously need tweaking with geological knowledge). The image shows that the core is dominated by 3 main mineralogical domains (Classes 3, 2, and 1) with small distinct “blobs” of class 4. But what are the classes? How different are they?

We can interrogate the cluster centres:

The spectra for Class 1 (blue line), Class 2 (orange line) and Class 3 (green line) are very similar. There are differences in intensity at shorter wavelengths (which explains the variation in the false colour image), but they all have the same features with varying depths. Class 4 (red line) is different, while it shares the shorter wavelength features as the other classes, it has a distinctly different feature around 2335nm.

Now that we know a little bit more about the spectral variation in the core, we can try to simply match these cluster centres to spectra from the Ecostress library.

Note, I didnt save the data, and when I came back to it I had to start from raw, thus a different mask and the clustering came out differently. Don’t be me, the save button is right there!

The direct library matches are giving us a mixture of usefully recognised minerals and nonsense. The matches have been performed using three different metrics; Pearson correlation, Spectral Angle Mapping and Modified Spectral Angle Mapping.

Class 4 is confidently identified as Calcite by two of the three metrics, and Classes 2 and 3 are identified as Montmorillonite. Class 1 is identified as elemental Sulfur by the Pearson correlation which is…unlikely. This class is clearly being matched on noise, and doesn’t reach the threshold for identification in the other two methods.

We can also just directly compare each individual spectra against the library directly, using any of the three techniques to produce a mineral map.

Again a mixture of usefully recognised minerals and nonsense.

Take-Away

In under 10 minutes, without writing code, we can open, correct and interpret hyperspectral core scanning data, easily!

I can confidently say that there is probably some carbonate in discrete areas and the rest is probably some kind of clay or mica, and the key scientific conclusion in everything ever done: Further work is needed.

To go further, and be more confident in your interpretations you will need to;

• examine the individual features (tools exist in CoreSpecViewer),
• curate the library to ensure matches are checked against realistic options (tools exist in CoreSpecViewer),
• use your own library (open it in CoreSpecViewer),
• select spectra from your data and build your own library (tools exist in CoreSpecViewer).

That’s the point: Hyperspectral interpretation still requires geologists, not just algorithms — but the barrier to entry shouldn’t be the software.

If you have hyperspectral core scanning data in raw Lumo output files, check out CoreSpecViewer and send me all your complaints!

If you have hyperspectral core scanning data from other acquisition methods, let’s see if we can collaborate to extend CoreSpecViewer to smooth file discovery and metadata parsing!

If you don’t have any data, but want to play with the software, there are links to some open source data on the GitHub landing page.

The Original Linkedin post.

Data and licence

The hyperspectral core scans used in this note (hole NWG-05-03, box 303) are published by the Alberta Geological Survey (AGS) and were downloaded from:

Alberta Geological Survey (2023): Core Data Interactive Map; Alberta Energy Regulator / Alberta Geological Survey, AER/AGS Interactive App or Map 014, https://ags.aer.ca/publications/all-publications/iam-014 [accessed August 2026].

The scanning programme is indexed in Spectrum Geosciences Ltd. and TerraCore Geospectral Imaging (2026): Index to hyperspectral core scanning imagery for mineral core from the Athabasca Basin, Canadian Shield, and Western Canada Sedimentary Basin in Alberta; Alberta Energy Regulator / Alberta Geological Survey, AER/AGS Special Report 129, 15 p.

Contains information licensed under the Open Government Licence – Alberta.