Kaggle S&P 500 Intraday Data Migration



Introducing the Kaggle S&P 500 Intraday Data Migration

1. Check raw data

Based on the image, it appears that preprocessing will be necessary.

2. Set the Preprocessing Plan

  • Modify multi-level column names to single column names.
  • Linear interpolation & Fill NaN values.
  • Convert to the format: Name, Time, Value.
  • Convert to UTC time.

Modify multi-level column names to single column names

Linear interpolation & Fill NaN values

Convert to the format: Name, Time, Value

Convert to UTC time

Once the data frame is restructured as shown in the image below, it will be ready for upload to Machbase Neo.

3. Data Upload

Finally, the data can be uploaded to Machbase Neo using the command below.

machbase-neo shell import --input ./datahub-2024-07-SP500.csv.gz --compress gzip --header --method append --timeformat ns sp500

Check the entire code.

datahub/dataset/2024/06.Electricity Transformer Dataset (ETDataset)/conv/convert.py at main · machbase/datahub
All Industrial IoT DataHub with data visualization and AI source - machbase/datahub

4. Check the results after uploading

Output when executing the following code in the Machbase Neo internal shell.

select * from v$sp500_stat;

※ Subsequent link to the AI training process: Kaggle S&P 500 Intraday Data

Back to Top