NEM Explorer is SCADA Miner’s natural-language interface for Australian National Electricity Market (NEM) data. It lets analysts, asset operators, developers, consultants, traders and technically curious energy people ask questions about AEMO prices, dispatch, generation mix, batteries, FCAS, constraints, interconnector flows and historical market events.
A critical aspect of the tool aside from making AEMO data easier to query is to make the answer auditable.
A NEM data answer can look plausible while still being wrong. The SQL may run. The chart may render. The number may be in the right order of magnitude. But if the query used the wrong interval, missed a deduplication rule, treated FCAS availability as FCAS enablement, or neglected to consider energy targets must follow a linear ramp, the result can still be materially wrong.
NEM Explorer is tested against external ground truths: published figures from sources such as the Australian Energy Regulator (AER), AEMO, the Clean Energy Council, Open Electricity and other public market references. A passing validation test should mean more than “the warehouse agreed with itself”. It should mean the system can reproduce a number that exists outside the application, with any methodological difference understood and documented.
NEM Explorer returns an answer, chart and SQL so the result can be inspected
External validation matters
AEMO’s public data is rich, but it is not a single clean spreadsheet. The data arrives through NEMWeb, MMSDM tables, market notice corrections, dispatch files, settlement-related publications and newer reporting streams such as FPP. Different datasets use different timesteps, timestamps, revision patterns and definitions.
External validation catches errors that ordinary unit tests often miss.
For example:
Counting negative price intervals against 5-minute dispatch prices when the published reference is based on 30-minute trading intervals can produce a result that is directionally right but badly inflated.
Estimating generator revenue from dispatch targets requires a linear ramp across the interval, not an assumption that the generator spends the entire dispatch interval at the dispatch interval’s energy dispatch target.
Battery revenue needs signed dispatch targets, because charging intervals are negative. Likewise gauging BESS activity by taking an average of the dispatch target will make it appear to have done almost nothing for the day if it has completed a full charge-discharge cycle.
FCAS revenue is based on enabled MW, not offered availability.
A NEM-wide FCAS price is not the same thing as a straight average across five regional prices.
AEMO tables often contain duplicate or superseded rows unless the query uses the right intervention, version or schema deduplication rule.
These details matter because NEM Explorer is intended for real market questions informing investment decisions, measuring policy impact or analysing bidding strategies.
NEM Explorer data validation
NEM Explorer validates both numbers and method.
The validation set includes questions like:
What was the volume-weighted average spot price in NSW for Q4 2025?
How many 30-minute trading intervals had negative prices in South Australia in Q4 2025?
What was NEM demand over a defined period, and why does it differ from AER annual consumption?
How much brown coal generation did the Clean Energy Australia report publish for calendar year 2025?
What adjusted dispatch price did AEMO publish after a market intervention or price revision?
What FPP usage or performance value did AEMO publish in a particular FPPRUN or FPPDAILY file?
Which battery earned the most from energy and FCAS over a period, and can the energy and FCAS components be reconciled?
Each grounded question records:
the external source;
the exact datapoint in that source;
the expected value and unit;
the acceptable tolerance;
the NEM time window;
the warehouse table and columns exercised by the question;
fault-finding notes explaining known methodological differences.
Example: validating AER spot price figures
The Australian Energy Regulator publishes market statistics charts and downloadable datasets, including monthly volume weighted average, maximum and minimum spot prices. That page states the volume weighted average is weighted against demand for electricity.
If NEM Explorer answers a volume-weighted price question, the query must join price to demand and apply the right weighting. A simple time average may be useful for a different question, but it is not necessarily representative of true energy cost.
The validation does not assume every difference is a bug. AER may publish a rounded quarterly number. NEM Explorer may calculate from 5-minute dispatch or trading data depending on the wording. The test tolerance is therefore set to catch a real methodology regression while allowing expected rounding and timestep differences.
Negative price counts are a good test because a wrong answer can be very wrong.
AER references for negative price intervals are based on trading interval data. If a query counts every 5-minute dispatch interval below zero instead, it can over-count by about six times. That is exactly the sort of error an AI-generated SQL layer must be guarded against.
The validation question is worded to require 30-minute trading intervals. The expected value is pinned to the AER reference. The test then catches whether the system selects the correct trading-grain table rather than a superficially similar dispatch-price table.
This is the general pattern: the question should test the market concept, not just a table name.
Example: demand and rooftop solar differences
Some validation differences are real and should remain.
For demand, NEM Explorer can calculate grid demand from AEMO dispatch regional demand. AER annual consumption figures may include behind-the-meter rooftop PV and may be reported on a financial-year basis. If a validation compares August to June warehouse data with an AER full-year number pro-rated to 11 months, the warehouse result should be lower.
That is not necessarily a failure. It is a definition difference:
NEM Explorer is measuring operational demand met from the grid.
AER annual consumption can include broader consumption effects.
A pro-rated annual reference assumes demand is evenly distributed through the year, which is not quite true.
The validation therefore records the expected bias and uses a tolerance that is narrow enough to catch a bug but wide enough to allow the known definition gap.
This is important for users. If someone asks “how much electricity did Victoria use?”, the answer depends on whether they mean grid-supplied operational demand, native demand including rooftop PV, or metered customer consumption. A robust tool needs to know the difference.
Example: Clean Energy Council generation mix
The Clean Energy Council’s Clean Energy Australia report provides another useful type of validation: annual generation by fuel category.
For example, a validation question can ask how much brown coal generated in Australia in calendar year 2025, in GWh, and compare the answer with the Clean Energy Australia report appendix. In the warehouse, that question exercises the daily fuel generation rollup, fuel-type mapping and date filtering.
This kind of check is valuable because fuel mapping is easy to get subtly wrong. A DUID can change registration details. A station can contain multiple units. Fuel categories need to be consistent across the public report and the warehouse semantic layer. A single published annual number is a good independent anchor.
Clean Energy Australia reports provide useful reference values against which NEM Explorer’s accuracy is validated
Revenue validation: the linear ramp
Energy revenue is one of the highest-value NEM Explorer question classes. It is also one of the easiest to calculate incorrectly.
For a 5-minute dispatch interval, the dispatch target at the end of the interval is not the average output across the interval. The better dispatch-target estimate is the trapezoidal rule:
This matters when a unit is ramping. A rectangular calculation using only TOTALCLEARED assumes the unit was at the end-of-interval target for the whole interval. That can overstate or understate energy during ramps and battery charge/discharge transitions.
For batteries, the calculation must also preserve sign. Positive dispatch is generation. Negative dispatch is charging. Charging at a positive price is negative energy revenue. Charging at a negative price can be positive from an arbitrage perspective. Treating battery dispatch as unsigned destroys the economics of the interval.
NEM Explorer’s revenue validation is designed to catch those mistakes before they become confident but wrong answers to questions like:
How much did Eraring BESS earn last month?
Break down Western Downs Battery revenue into energy and FCAS.
Which batteries earned the most revenue per registered MW in March 2026?
How did energy arbitrage compare with FCAS revenue during a price event?
FCAS revenue: enablement is not availability
FCAS is another area where plausible wording can lead to wrong numbers.
For per-DUID FCAS revenue, the relevant quantity is enabled MW for the service in the dispatch interval. FCAS revenue is:
The 5 / 60 factor matters because FCAS prices are hourly prices and dispatch intervals are 5 minutes.
FCAS actual availability columns cannot be used as a revenue proxy. Availability is a potential capability measure. Enablement is the awarded MW. Availability is often greater than enablement and can materially inflate a revenue estimate.
This distinction is especially important for batteries, because battery revenue stacks often combine energy arbitrage and multiple FCAS services. A small error in service selection or enablement logic can change the commercial interpretation of the result.
AER Industry Charts provide a source of truth independent from the raw AEMO data
FPP validation: sign conventions and source rows
Frequency Performance Payments add a different kind of challenge.
FPP is a causer-pays style settlement mechanism. The important quantity is not just “did a unit deviate?” but whether the deviation helped or hurt frequency relative to the reference trajectory. NEM Explorer therefore needs concept knowledge as well as table knowledge.
The public AEMO FPPRUN and FPPDAILY files provide direct validation anchors. A test can pin to a published usage, performance or daily FPP value and then verify that the warehouse returns the same number.
A few example common issues unguided AI agents can make:
FPP cost columns are signed. A negative source value can mean the unit paid.
User-facing “cost” rankings usually need to present the amount paid as positive.
FPP usage and performance are not the same as FCAS enablement.
Some FPP tables require schema-aware deduplication before aggregation.
FPP financial settlement has a start date, so earlier indicative rows must be handled carefully.
These are exactly the details that disappear if validation only checks that an SQL query returns rows.
AEMO data clean-up: deduplication is part of correctness
AEMO data often needs mechanical cleanup before it is safe to query.
Common examples include:
filtering INTERVENTION = 0 on dispatch tables so intervention runs do not duplicate normal dispatch;
using the latest version when MMSDM tables publish multiple VERSIONNO rows;
folding AEMO price revisions into the dispatch price surface so adjusted prices are used after market notices or correction reports;
deduplicating FPP usage rows where schema versions overlap;
excluding settlement residue virtual units from physical generator fleet rankings;
joining DUIDs to region and station reference tables rather than inferring those facts from strings.
NEM Explorer’s semantic tables and capability layer encode these rules so common questions do not need to rediscover them each time. The validation suite then checks that those rules keep working as data sources, schemas and AI routing logic evolve.
Robust approach to validation
The validation approach is robust because it tests the full path from natural-language question to market answer:
The question is phrased like a user would ask it.
The system must select the right concept, table and time grain.
The generated SQL must apply the right market rule.
The answer is compared with an external published source.
Any accepted difference must have a written reason.
This catches classes of failures including:
wrong table selection;
wrong interval grain;
missing time scaling;
missing intervention or version deduplication;
incorrect DUID-to-region mapping;
FCAS enablement/availability confusion;
FPP sign errors;
fuel mapping drift;
prompt or routing regressions where an AI answer chooses the wrong capability.
It also gives a practical way to improve coverage. The goal is not to test every column in the warehouse. The goal is to ground the customer-facing query surface: prices, demand, dispatch, generation mix, revenue, FCAS, constraints, interconnectors, FPP and the reference data needed to join them correctly.
What users should expect
NEM Explorer is designed for questions such as:
What were the 10 most profitable wind farms by registered MW in March 2026?
Which regions had negative prices for more than 30% of the time last week?
Break down battery revenue across energy and FCAS markets.
Which FCAS service had the highest average price last quarter, and in which region?
Which unit had the highest FPP cost last month?
What constraints were binding during a price separation event?
Show interconnector flows between NSW and QLD during a high-price interval.
For each answer, the useful question is not only “what number did it return?” but “what definition did it use?”
That is why NEM Explorer exposes the SQL behind the answer and why its tests are grounded against public sources. In the NEM, definitions matter. Dispatch versus trading intervals, metered versus dispatch-target energy, enablement versus availability, mainland versus Tasmania, signed versus unsigned battery flow, and revised versus original prices can all change the answer.
The validation work is core product infrastructure. If NEM Explorer is going to help people investigate the Australian electricity market quickly, it must also help them trust, inspect and challenge the result.
