使用网络数据进行能源统计：方法论和关键经验.pdf

1、1Using Web Data for energy statistics:Methodology and key lessonsMathieu SCHEFFERMarianne DOULSandrine HERBETHWIN Conference|Gdansk,Poland|5 February 2025Introduction to the projectWhy Web Data?Growing need for timely,granular,and comprehensive statistics.Web datas potential to complement traditiona

2、l datasets.Project Overview:Tapping New Data Sources awarded by Eurostat.Focus:Integrating web-based and alternative data for energy statistics.Objectives:Enhance timeliness and relevance of energy statistics.Develop innovative methodologies for web data extraction and validation.WIN Conference|Gdan

3、sk,Poland|5 February 2025Identification of data sourcesReliable,relevant,and accessibleAll energy products and energy prices71 institutions/associations websites69 national websites 140 websites identifiedCountry coverageData qualityTime coverageEnergy products presented in the indicators,or pricesA

4、dditionialcriteria18 websitesENTSO-E,ENTSO-G,GIE,the Energy Institute and EurofuelWIN Conference|Gdansk,Poland|5 February 2025Tool developments Platforms analysis and needsWIN Conference|Gdansk,Poland|5 February 2025Tool developments-PYTHONLibraries used:Steamlit and BeautifulSoupConfigurable tool:L

5、ist of countriesList of indicatorsTime rangesAPI interactionsExtraction into CSV formatWIN Conference|Gdansk,Poland|5 February 2025Tool developments Each platform its own set of requirementsWIN Conference|Gdansk,Poland|5 February 2025both required direct database access via API.ENTSO-E and ENTSO-G p

6、rovided an Excel file,so we built a feature that allows users to select only the needed indicators.The Energy Institute required API access,but the selection process differed slightlyGIETechnical challengesAPI limitationsProxy restrictionsConfidentiality,acces