San Diego Daily Weather Data Migration



Introducing the San Diego Daily Weather Data Migration

1. Check raw data

The handling of missing values and duplicate values seems necessary.

2. Set the Preprocessing Plan

  • Handling duplicate values using Resampleing at 1-minute intervals.
  • Handling missing values using Linear Interpolation.
  • Convert to the format: Name, Time, Value & Convert to UTC time.

Handling duplicate values using Resampleing at 1-minute intervals

Since the original data is in 1-minute intervals, duplicate values are handled by resampling at 1-minute intervals.

Handling missing values using Linear Interpolation

Convert to the format: Name, Time, Value & Convert to UTC time

Once the data frame is restructured as shown in the image below, it will be ready for upload to Machbase Neo.

3. Data Upload

Finally, the data can be uploaded to Machbase Neo using the command below.

machbase-neo shell import --input ./datahub-2024-14-San-Diego-Daily-Weather.csv.gz --compress gzip --header --method append --timeformat ns san_diego_weather

Check the entire code.

datahub/dataset/2024/14.San Diego Weather/conv/convert.py at main · machbase/datahub
All Industrial IoT DataHub with data visualization and AI source - machbase/datahub

4. Check the results after uploading

Output when executing the following code in the Machbase Neo internal shell.

select * from v$san_diego_weather_stat;

※ Subsequent link to the AI training process: San Diego Daily Weather Data

Back to Top